Camera intrinsic parameter calibration method and device, and electronic equipment

Abstract

The application discloses a camera intrinsic parameter calibration method and device and electronic equipment, and the method comprises the steps of collecting an image in a current scene, identifying the boundary pixel points of the linear structure with distortion in the collected image, and obtaining the boundary position information of the linear structure; sampling the boundary of the linear structure in the horizontal direction and the vertical direction according to the boundary position information, and fitting the sampling points to obtain the mean position of the boundary as the distortion center in the horizontal direction and the vertical direction; constructing an adaptive function based on the sum of squares of errors, and calculating the global optimal solution of the distortion coefficient according to the adaptive function; extracting parallel line identification information in the current scene, and obtaining the focal length closed solution based on the relative position relationship of the parallel lines. According to the above technical scheme, the calibration process is simplified, the camera intrinsic parameter calibration is completed only by relying on the linear features in the road scene, the camera can be adaptively calibrated in a dynamic scene, and the accuracy and stability of the calibration are effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of communication, in particular to a camera intrinsic parameter calibration method and device and electronic equipment. BACKGROUND

[0002] With the rapid development of intelligent driving technology, the accurate perception and decision-making ability of the automatic driving system has become the core demand to realize safe and stable driving. The stereo vision system as an important visual sensing method can obtain the distance information of the scene through two cameras, providing the three-dimensional structure of the vehicle surrounding environment for the autonomous vehicle, so that it has the distance understanding of the environment, especially in the complex traffic environment. In the scene of highway, urban road, etc., precise environment modeling and analysis using line features (such as lane lines, road edge lines, traffic signs, etc.) are crucial for positioning, path planning and obstacle avoidance of autonomous vehicles.

[0003] In related technologies, camera intrinsic parameter calibration mainly depends on a physical calibration board. However, the calibration depending on the physical calibration board has certain limitations, mainly reflected in the complex operation process, high environmental requirements, and relatively large maintenance cost.

[0004] Therefore, it is an urgent problem to be solved to provide a simple and easy-to-implement camera intrinsic parameter calibration scheme that does not depend on a physical calibration board. SUMMARY

[0005] The main purpose of the present application is to disclose a camera intrinsic parameter calibration method, device and electronic equipment, to at least solve the problems of complex operation process, high environmental requirements, and relatively large maintenance cost in the related art camera intrinsic parameter calibration scheme depending on a physical calibration board.

[0006] According to one aspect of the present application, a camera intrinsic parameter calibration method is provided.

[0007] The camera intrinsic parameter calibration method provided by the present application comprises: collecting an image in a current scene, identifying the boundary pixel points of a linear structure with distortion in the collected image, and obtaining the boundary position information of the linear structure; sampling the boundaries of the linear structure in the horizontal direction and the vertical direction according to the boundary position information, and fitting the sampling points to obtain the mean position of the boundary as the distortion center in the horizontal direction and the vertical direction; constructing an fitness function based on the sum of squares of errors, and calculating the global optimal solution of distortion coefficients according to the fitness function; extracting parallel line identification information in the current scene, and obtaining a focal length closed solution based on the relative position relationship of the parallel lines.

[0008] Further, the boundary pixel points of the linear structure with distortion in the collected image are identified, and the boundary position information of the linear structure is obtained, including: using a semantic segmentation structure of a Mobile-Seed network to identify a region of the linear structure with distortion in the collected image; using a boundary detection structure of the Mobile-Seed network to extract boundary pixel points of the linear structure with distortion in the collected image, and obtain the boundary position information of the linear structure.

[0009] Further, the boundary of the linear structure in the horizontal direction and the vertical direction is sampled according to the boundary position information, and the mean position of the boundary is fitted to obtain the distortion center in the horizontal direction and the vertical direction, including: sampling the boundary of the linear structure in the horizontal direction and the vertical direction to obtain a plurality of discrete sampling points as input data for Gaussian curve fitting; and obtaining the mean position of the distorted curve of the linear structure by performing Gaussian curve fitting on the sampling points, as the distortion center in the horizontal direction and the vertical direction.

[0010] Further, the fitness function is constructed based on the sum of squares of errors, and the global optimal solution of the distortion coefficient is calculated and obtained according to the fitness function, including:

[0011] The fitness function Fitness is constructed by the following formula:

[0012]

[0013] Wherein, is the number of sampling distortion-corrected image points, is the i-th distortion-corrected de-distortion point coordinate, is a straight line obtained by least squares fitting of the de-distortion point coordinates, is the slope of the fitted straight line, is the intercept of the fitted straight line;

[0014] In the case where the value of the fitness function Fitness is minimum, the radial distortion coefficients c1, c2 and the tangential distortion coefficients d1, d2 are taken as the global optimal solution of the distortion coefficients.

[0015] Further, in the case where the value of the fitness function Fitness is minimum, the radial distortion coefficients c1, c2 and the tangential distortion coefficients d1, d2 are taken as the global optimal solution of the distortion coefficients, including:

[0016] S1: The sampling points obtained by sampling the boundary of the linear structure with distortion in the image are subjected to distortion correction by applying randomly initialized radial distortion coefficients c1, c2 and tangential distortion coefficients d1, d2;

[0017] S2: for the sampling points after distortion correction, the straight line obtained by least square fitting the sampling point coordinates ;

[0018] S3: substituting the coordinates of the n sampling points after distortion correction, the values of m and b into the fitness function formula, to calculate the value of the fitness function;

[0019] S4: cyclically executing the S1 to S3, each time dynamically adjusting the c1, c2, d1, d2, through continuous iterative search, in the case of meeting the value of the fitness function Fitness being minimum, the corresponding radial distortion coefficients c1, c2 and the tangential distortion coefficients d1, d2 are taken as the global optimal solution of the distortion coefficients.

[0020] Further, the focal length closed solution is obtained based on the relative position relationship of the parallel lines, including: extracting one or more groups of parallel lines in the current scene according to a specific mark; for each group of parallel lines, obtaining a unique solution of the projection depth corresponding to the multiple end points of the group of parallel lines according to the image point coordinates corresponding to the multiple end points of the group of parallel lines; obtaining the focal length closed solution corresponding to the group of parallel lines according to the unique solution of the projection depth, the distortion center and the image point coordinates; determining the final focal length closed solution by synthesizing the calculation results of the focal length closed solutions corresponding to the one or more groups of parallel lines.

[0021] Further, for each group of parallel lines, obtaining a unique solution of the projection depth corresponding to the multiple end points of the group of parallel lines according to the image point coordinates corresponding to the multiple end points of the group of parallel lines includes: constructing a binocular stereo vision system, setting the origin of the world coordinate system at the optical center of one of the binoculars; substituting the coordinates of the multiple end points of the group of parallel lines in the world coordinate system and the image point coordinates corresponding to the multiple end points into the projection equation; finding the correlation between the projection depth and the image point coordinates according to the projection equation to obtain a constraint equation; setting the projection depth to be positive and all effective three-dimensional points in the image to be located in front of the camera, in the case that the image point coordinates are known, obtaining a unique solution of the projection depth corresponding to the multiple end points by solving the zero space of the matrix composed of the image point coordinates.

[0022] Further, obtaining the focal length closed solution corresponding to the group of parallel lines according to the unique solution of the projection depth, the distortion center position and the image point coordinates includes:

[0023] The focal length closed solution f is obtained by the following formula:

[0024]

[0025] wherein, , and are the first, second, third, fourth, fifth, sixth, seventh, eighth, ninth, tenth, eleventh, twelfth, thirteenth, fourteenth, fifteenth, sixteenth, seventeenth, eighteenth, nineteenth, twentieth, twenty-first, twenty-second, twenty-third, twenty-fourth, twenty-fifth, twenty-sixth, twenty-seventh, twenty-eighth, twenty-ninth, thirtieth, thirty-first, thirty-second, thirty-third, thirty-fourth, thirty-fifth, thirty-sixth, thirty-seventh, thirty-eighth, thirty-ninth, and fortieth elements of the vector and is the distortion center, are the epipolar depths of the two end points of the line segment in the set of parallel lines, respectively, are the epipolar depths of the two end points of another line segment in the set of parallel lines which is equal in length to the line segment, respectively, are the image point coordinates corresponding to the two end points of the line segment, and are the image point coordinates corresponding to the two end points of the another line segment.

[0026] According to another aspect of the present application, there is provided a camera intrinsic parameter calibration device.

[0027] The camera intrinsic parameter calibration device according to the present application comprises: an identification module, configured to collect an image in a current scene, identify the boundary pixel points of a linear structure with distortion in the collected image, and obtain the boundary position information of the linear structure; a distortion center acquisition module, configured to sample the boundaries of the linear structure in the horizontal direction and the vertical direction according to the boundary position information, and obtain the mean position of the boundaries by fitting the sampling points, as the distortion center positions in the horizontal direction and the vertical direction; a distortion coefficient acquisition module, configured to construct an fitness function based on the sum of squares of errors, and calculate the global optimal solution of the distortion coefficient according to the fitness function; and a focal length acquisition module, configured to extract parallel line identification information in the current scene, and obtain the focal length closed solution based on the relative position relationship of the parallel lines.

[0028] According to still another aspect of the present application, there is provided an electronic device.

[0029] The electronic device according to the present application comprises a processor and a memory, wherein the memory is configured to store executable instructions of the processor, and the processor is configured to execute the camera intrinsic parameter calibration method according to any one of the above aspects by executing the executable instructions.

[0030] According to the present application, a camera intrinsic parameter calibration method without a physical calibration board is provided, which directly utilizes the linear structures (e.g., road markings, building edges, etc.) in the current scene (e.g., a road scene) for distortion calibration by collecting an image in the current scene, effectively solving the problem of dependence on a calibration board in the traditional method, and the operation process is simple and easy to implement, the environmental requirements are not high, and the maintenance cost is relatively low. BRIEF DESCRIPTION OF DRAWINGS

[0031] ​​​​In order to make the technical solutions in the prior art or the embodiments of the present application clearer, the accompanying drawings needed in the description of the embodiments or the prior art will be briefly introduced. Obviously, the accompanying drawings in the following description are only exemplary and, for those skilled in the art, other drawings can be obtained from the provided drawings without any creative effort.

[0032] Figure 1 is a flow chart of a camera intrinsic parameter calibration method according to an embodiment of the present application;

[0033] Figure 2 is a schematic diagram of center point calibration according to an embodiment of the present application;

[0034] Figure 3 is a schematic diagram of obtaining focal length through parallel lines according to an embodiment of the present application;

[0035] Figure 4 is a flow chart of a camera intrinsic parameter calibration method according to a preferred embodiment of the present application;

[0036] Figure 5 is an undistorted effect diagram according to an embodiment of the present application;

[0037] Figure 6 is a structural block diagram of a camera intrinsic parameter calibration device according to an embodiment of the present application;

[0038] Figure 7 is a structural block diagram of an electronic device according to a preferred embodiment of the present application. DETAILED DESCRIPTION

[0039] The embodiments of the present application will be described in detail by the specific, concrete embodiments, and those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in the description. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort fall within the scope of protection of the present application.

[0040] The specific implementation of the present application will be described in detail below with reference to the accompanying drawings of the description.

[0041] According to an embodiment of the present application, a camera intrinsic parameter calibration method is provided.

[0042] Figure 1 is a flow chart of a camera intrinsic parameter calibration method according to an embodiment of the present application. As shown in Figure 1 , the camera intrinsic parameter calibration method comprises:

[0043] Step S101: Collect an image in the current scene, identify the boundary pixel points of the linear structure with distortion in the collected image, and obtain the boundary position information of the linear structure;

[0044] Step S102: Sample the boundaries of the linear structure in the horizontal and vertical directions according to the boundary position information, and fit the sampling points to obtain the mean position of the boundary as the distortion center in the horizontal and vertical directions;

[0045] Step S103: Construct a fitness function based on the sum of squares of errors, and calculate the global optimal solution of the distortion coefficient according to the fitness function;

[0046] Step S104: Extract parallel line identification information in the current scene, and obtain a focal length closed solution based on the relative position relationship of the parallel lines.

[0047] The camera intrinsic parameter calibration scheme in the related art depends on a physical calibration board, and has problems such as complex operation process, high environmental requirement, and relatively large maintenance cost. Figure 1 The camera intrinsic parameter calibration method provided by the application collects an image in the current scene, and calibrates distortion by directly using the linear structure (such as road markings, building edges, etc.) in the current scene (such as a road scene, etc.), which effectively solves the problem of dependence on a calibration board in the related art. Figure 1 The camera intrinsic parameter calibration method provided by the application simplifies the calibration process, and only relies on the linear features in the road scene to complete the camera intrinsic parameter calibration, so that the camera can be adaptively calibrated in a dynamic scene, and the accuracy and stability of the calibration are effectively improved.

[0048] It should be noted that the above-mentioned camera intrinsic parameter refers to a parameter describing the geometry and optical characteristics of the camera itself, and usually includes focal length, principal point position (usually coinciding with the distortion center), distortion coefficient, etc.

[0049] Among them, the distortion coefficient is used to describe and correct the image distortion caused by the camera lens, and common distortion types include radial distortion and tangential distortion. Radial distortion is caused by the spherical shape of the lens, and the points in the center of the image and the points at the edge of the image are enlarged or reduced to different degrees. Tangential distortion is caused by imperfect alignment or installation between the lens and the image sensor, and is manifested as slight skew of the image;

[0050] Among them, the distortion center refers to the area in the image where the distortion is the strongest, and is usually located at the center of the image. Most camera lenses have radial distortion and tangential distortion, and the distortion center is the area where the distortion is most significant, and usually coincides with the optical center (principal point position) of the camera.

[0051] Wherein, focal length, refers to the distance from the optical center (focal point) of the lens to the imaging plane, the focal length determines the camera's viewing angle and magnification.

[0052] In step S101, the boundary pixel points of the linear structure with distortion in the collected image are identified, and the boundary position information of the linear structure can further include the following processing: using the semantic segmentation structure of the Mobile-Seed network to identify the area of the linear structure with distortion in the collected image; using the boundary detection structure of the Mobile-Seed network to extract the boundary pixel points of the linear structure with distortion in the collected image, and obtaining the boundary position information of the linear structure.

[0053] In the preferred embodiment, first, the Mobile-Seed network can be used to accurately identify the boundaries of linear structures (such as road markings, lamp posts, etc.) with distortion in the original image, and obtain the boundary position information of these straight line structures that are curved due to lens distortion. The above Mobile-Seed network achieves this goal by combining the dual-flow structure of semantic segmentation and boundary detection. Among them, the semantic flow is responsible for global semantic segmentation, identifying the linear structure area in the image, while the boundary flow focuses on accurately detecting the boundaries of these structures, outputting a boundary map containing the pixel position information of the linear structure boundaries in the image. By dynamically adjusting the feature fusion weight of the semantic flow and the boundary flow, this method can adaptively optimize the feature fusion of different image regions, thereby significantly improving the boundary recognition accuracy and robustness in distorted images.

[0054] In the above step S102, the boundaries of the linear structure in the horizontal and vertical directions are sampled according to the boundary position information, and the mean position of the boundary is obtained by fitting the sampling points as the distortion center in the horizontal and vertical directions, which can further include the following: sampling the boundaries of the linear structure in the horizontal and vertical directions to obtain a plurality of discrete sampling points as input data for Gaussian curve fitting; obtaining the mean position of the distorted curve of the linear structure by performing Gaussian curve fitting on the sampling points as the distortion center in the horizontal and vertical directions.

[0055] According to the property of distortion center symmetry, before distortion correction, the straight line in the image will be distorted and appear as a curve. As shown in Figure 2 Due to the distortion characteristics of the camera lens, the straight line without distortion and appears as a curve and However, these curves still maintain the central symmetry property. Based on this property, the boundary position information of the object (e.g., road marking, building edge, etc.) with distortion accurately extracted by the Mobile-Seed network is used to perform sampling (usually uniform sampling, but also non-uniform sampling) on the identified boundary to obtain a series of discrete points as the input data for Gaussian curve fitting.

[0056] The mean position of the curve is obtained by fitting the sampling points of the linear structure with distortion in the horizontal and vertical directions of the image, respectively, as a key parameter in camera intrinsic parameter calibration The camera intrinsic parameter matrix is expressed as:

[0057]

[0058] wherein, and The calculation formula of , represents the focal length, and represent the pixel size in the direction and the direction, is the distortion center.

[0059] It can be seen that, by fusing the curve recognition and Gaussian curve fitting technology of the image, the distortion center is positioned based on the linear structure (such as road marking, lamp post, etc.) with distortion collected in the original image in the current scene (for example, road scene). By calculating the mean value of a plurality of pairs of distortion curve fitting results, the positions of the distortion centers in the horizontal and vertical directions are finally obtained as key parameters in intrinsic parameter calibration.

[0060] In step S103, the fitness function is constructed based on the sum of squares of errors, and the global optimal solution of the distortion coefficient is calculated and obtained according to the fitness function, which can further include the following processing:

[0061] Firstly, the fitness function Fitness is constructed by the following formula:

[0062]

[0063] wherein, is the number of sampling distortion-corrected image points, is the i-th distortion-corrected de-distortion point coordinate, is a straight line obtained by least square fitting of the de-distortion point coordinates, is the slope of the fitted straight line, is the intercept of the fitted straight line;

[0064] Second, in the case of the minimum value of the fitness function Fitness, the radial distortion coefficients c1, c2 and the tangential distortion coefficients d1, d2 are taken as the global optimal solution of the distortion coefficients.

[0065] Wherein, in the case of the minimum value of the fitness function Fitness, the radial distortion coefficients c1, c2 and the tangential distortion coefficients d1, d2 are taken as the global optimal solution of the distortion coefficients can further include the following processing:

[0066] S1: the sampling points obtained by sampling the boundaries of the linear structures in the image with distortion are subjected to distortion correction by applying randomly initialized radial distortion coefficients c1, c2 and tangential distortion coefficients d1, d2;

[0067] S2: the straight lines obtained by least square fitting the coordinates of the sampling points after distortion correction ;

[0068] S3: the values of the coordinates of the n sampling points after distortion correction, m and b are substituted into the fitness function formula to calculate the value of the fitness function;

[0069] S4: the S1 to S3 are executed in a loop, and c1, c2, d1, d2 are dynamically adjusted each time, and through continuous iterative search, the corresponding radial distortion coefficients c1, c2 and the tangential distortion coefficients d1, d2 are taken as the global optimal solution of the distortion coefficients in the case of satisfying the minimum value of the fitness function Fitness.

[0070] Wherein, the above-mentioned distortion coefficients are used to compensate the image distortion caused by lens distortion, and the curve is corrected back to its original straight line form. The above-mentioned fitness function Fitness is a function used to evaluate the individual in genetic algorithm. Its function is to measure the fitness of the individual according to its performance in a specific task. Specifically, the fitness function returns a numerical value indicating the degree of excellence of the individual (solution). The optimization goal of the fitness function is usually to maximize or minimize a certain index.

[0071] In the preferred implementation process, based on the distortion center positioning result, the fitness function Fitness based on the sum of squares of errors, and the global optimization is carried out by combining the genetic algorithm to solve the distortion coefficients, so as to ensure that the optimization result of the distortion coefficients reaches the global optimum. Wherein, the above-mentioned genetic algorithm refers to an optimization algorithm simulating the principles of natural selection and genetics, which solves the optimization problem by simulating the process of biological evolution in nature, and is suitable for solving complex or high-dimensional optimization problems, especially the problems that cannot be solved by traditional methods. First, according to the geometric restoration principle of the distortion coefficients, the known sampling points on the boundaries of the distorted objects in the image can be subjected to distortion correction by applying randomly initialized radial distortion coefficients and tangential distortion coefficients The distortion correction is performed, and the correction formula of the distortion coefficient for the distorted image point is as follows:

[0072]

[0073] wherein, denotes the coordinates of a point in the image after distortion correction, denotes the coordinates of the point in the image with distortion. is the square of the distance of the point to the distortion center .

[0074] It should be noted that when the distortion coefficient is closer to the true value, the fitting result of the restored sampling point will be closer to a straight line, and therefore, the straightness of the curve reflects the degree to which the distortion parameter is close to the true value. Based on this characteristic, the above fitness function is designed by evaluating the fitting degree of the coordinate point after distortion correction and the ideal straight line to measure the quality of the distortion coefficient. In order to achieve this goal, the fitness function is defined as follows:

[0075]

[0076] wherein, is the number of sampling image points after distortion correction, is the i-th corrected distortion-free point coordinate, is a straight line obtained by least square fitting of the distortion-free point coordinates, is the slope of the fitted straight line, is the intercept of the fitted straight line.

[0077] In the optimization process of the genetic algorithm, the smaller the value of the fitness function, the more effectively the current distortion coefficient can correct the image distortion, so that the fitted straight line is closer to the ideal distortion-free straight line, and the error is smaller. Such distortion coefficient will obtain a higher fitness, and thus be preferentially retained and optimized in the iteration. Through continuous iterative search, the algorithm finally converges to the optimal solution of the distortion coefficient, thereby accurately estimating the radial distortion coefficient and the tangential distortion coefficient , and achieving accurate correction of the image distortion.

[0078] In step S104, the parallel line identification information is extracted in the current scene, and the focal length closed solution is obtained based on the relative position relationship of the parallel lines. The obtaining of the focal length closed solution can further include the following processing: a group or multiple groups of parallel lines are extracted according to specific identification in the current scene; for each group of parallel lines, a projection depth unique solution corresponding to multiple end points of the group of parallel lines is obtained according to image point coordinates corresponding to the multiple end points; a focal length closed solution corresponding to the group of parallel lines is obtained according to the projection depth unique solution, the distortion center and the image point coordinates; and a final focal length closed solution is determined by synthesizing calculation results of the focal length closed solutions corresponding to the one or more groups of parallel lines.

[0079] wherein the projection depth represents distance information of an object in a camera image, and indicates the distance of each point in a scene from the camera.

[0080] wherein the obtaining of the projection depth unique solution corresponding to the multiple end points of the group of parallel lines can further include the following processing: a binocular stereo vision system is constructed, and the origin of a world coordinate system is set to be located at the optical center of one of the binoculars; coordinates of the multiple end points of the group of parallel lines in the world coordinate system and image point coordinates corresponding to the multiple end points are substituted into a projection equation; a constraint equation is obtained by searching for a correlation between the projection depth and the image point coordinates according to the projection equation; the projection depth is set to be positive, and all effective three-dimensional points in the image are set to be located in front of the camera; and a unique solution of the projection depth corresponding to the multiple end points is obtained by solving a zero space of a matrix composed of the image point coordinates, under the condition that the image point coordinates are known.

[0081] wherein the focal length closed solution f is obtained according to the projection depth unique solution, the distortion center position and the image point coordinates by the following formula:

[0082]

[0083] wherein, , and are the first elements of vectors and , , , is the distortion center, , are projection depths of two end points of a line segment in the group of parallel lines, are projection depths of two end points of another line segment in the group of parallel lines which is equal in length to the line segment, The coordinates of the image points corresponding to the two endpoints of the line segment are given. The coordinates of the image points corresponding to the two endpoints of the other line segment are given.

[0084] The following combination Figure 3 The examples further illustrate the preferred embodiments described above.

[0085] like Figure 3 As shown, the world coordinate system is defined as follows: The coordinate systems of the left and right cameras are defined as follows: Set the origin of the world coordinate system. Located in the left eye center (Of course, you can also set the origin of the world coordinate system.) Located at the center of right eye (center of view). In the current scene, extract one or more sets of parallel lines based on specific markers (e.g., zebra crossings), where, among the extracted parallel lines, the first... A pair of endpoints of a line segment are Their corresponding points in the image are Define the coordinates of a point in space as... The coordinates of the corresponding image point are The projective equation is expressed as:

[0086]

[0087] in, This represents the camera intrinsic parameter matrix. This is a scaling factor to ensure that the final image point coordinates can be correctly normalized. and These are the rotation matrix and the translation vector, respectively.

[0088] In a set of parallel lines, extract parallel line segments of equal length. The four endpoints of these parallel line segments are... , These are the coordinates of the corresponding image points. Substituting the endpoint coordinates into the projective equation yields:

[0089]

[0090] in, Corresponding points The depth of projection, Corresponding points The projection depth.

[0091] Since the coordinates of points in the world coordinate system are unknown, by subtracting the above four equations pairwise to simplify the equations and eliminate common terms, we can find the direct relationship between the projection depth and the coordinates of image points, and obtain the following constraint equations:

[0092]

[0093] in, ,

[0094] .

[0095] To resolve scale ambiguity between variables, the projective depth is always set to positive, and all valid 3D points in the image are located in front of the camera. This is because the image point coordinates... It is known that a unique solution to the projective depth can be obtained by solving the null space of a system of linear equations. The specific calculation formula is as follows:

[0096]

[0097] in, This represents a matrix consisting of the coordinates of image points, in the form of... , Representation matrix Zero space, The function is used to ensure that the returned value is an absolute value, thereby guaranteeing that the projection depth is positive, and thus uniquely solving for the projection depth.

[0098] Based on the above projective equation, by finding the direct relationship between the projective depth and the coordinates of the image points, the constraint equations obtained can be summarized into the following two equations:

[0099]

[0100] For specific landmarks in the scene (e.g., zebra crossings), their direction vectors satisfy the following condition: Furthermore, we define: , All are 3×1 vectors. Based on the orthogonality condition of the rotation matrix... The following equation is obtained:

[0101]

[0102] To simplify computational complexity, the following assumptions are made: Based on the above equation, with only the focal length unknown, the focal length is obtained. The closed solution is:

[0103]

[0104] in, , and They are vectors and The One element, , is the principal point position (i.e., the distortion center described above).

[0105] In the preferred implementation, in view of the limited precision problem existing in focal length calibration for single set of parallel line segment geometric constraints, the application provides a focal length calibration method based on multiple sets of parallel line segments. The method calculates the focal length of multiple sets of parallel line segments and observes the change trend of the focal length value with the increase in the number of sets. When the focal length value tends to be stable and the change range is very small, for example, the deviation of each focal length closed solution from the mean value does not exceed a preset threshold, the final focal length closed solution is determined. Then, the intrinsic parameter calibration result is used for image de-distortion processing.

[0106] The preferred implementation is described below in combination with Figure 4 The preferred implementation is described below in combination with

[0107] Figure 4 is a flowchart of a camera intrinsic parameter calibration method according to the preferred embodiment of the application. As shown in Figure 4 , the camera intrinsic parameter calibration method comprises the following steps:

[0108] Step S401: Collect an image in the current scene, identify the boundary pixel points of the linear structure (curved curve) with distortion in the collected image, and obtain the boundary position information of the linear structure. Specifically, the Mobile-Seed network can be used to extract the boundary pixel points of the curved curve in the image.

[0109] Step S402: Analyze the uniformly sampled points on the boundary using Gaussian curve fitting to accurately locate the distortion center. Specifically, by fusing the curve recognition and Gaussian curve fitting techniques of the image, the distortion center is located based on the linear structure (such as road markings, lamp posts, etc.) with distortion in the collected original image. By calculating the mean value of multiple pairs of distortion curve fitting results, the distortion center positions in the horizontal and vertical directions are finally obtained as the key parameters in the intrinsic parameter calibration.

[0110] Step S403: Based on the distortion center positioning result, design a fitness function based on the sum of squared errors, and combine a genetic algorithm for global optimization to solve the global optimal solution of the distortion coefficient, ensuring that the optimization result of the distortion coefficient reaches the global optimum. Step S404: Extract the parallel line identification information in the current scene, and use the parallel line property to analyze the relative position relationship of the parallel lines in space, and propose a focal length solution based on parallel line constraints to calculate the focal length closed solution. The focal length solution uses the projective geometry theory combined with the parallel line features in the image to derive the closed solution of the focal length.

[0111] In the preferred implementation process, in view of the limited accuracy problem existing in focal length calibration for single set of parallel line segment geometric constraints, a focal length calibration method based on multiple sets of parallel line segments is proposed. The method calculates the focal length of multiple sets of parallel line segments, and observes the change trend of the focal length value with the increase of the number of sets. When the focal length value tends to be stable and the change range is very small, the final intrinsic parameter calibration result is determined.

[0112] Step S405: image de-distortion processing is performed using the intrinsic parameter calibration result.

[0113] Through the above embodiment, in view of the fact that the complexity of the scene has a crucial influence on the camera intrinsic parameter calibration result, the embodiment combines line feature recognition technology and parallel line constraints to provide a solution without additional physical calibration equipment. This method improves the flexibility and adaptability of calibration while achieving real-time execution in the actual running environment, simplifying the calibration process. Moreover, image de-distortion processing is performed using the intrinsic parameter calibration result, and the de-distortion effect diagram is as shown in Figure 5 By comparing the images before and after de-distortion, it can be seen that the image after the calibration parameter processing is effectively corrected, and the image quality is significantly improved, further verifying the effectiveness of the technical solution provided by the present application in practical application.

[0114] According to the embodiment of the present application, a camera intrinsic parameter calibration device is provided.

[0115] Figure 6 is a structural block diagram of the camera intrinsic parameter calibration device according to the embodiment of the present application. As shown in Figure 6 The camera intrinsic parameter calibration device includes: an identification module 60, configured to collect images in the current scene, identify the boundary pixel points of the linear structure with distortion in the collected images, and obtain the boundary position information of the linear structure; a distortion center acquisition module 62, configured to sample the boundaries of the linear structure in the horizontal direction and the vertical direction according to the boundary position information, and fit the sampling points to obtain the mean position of the boundaries as the distortion center position in the horizontal direction and the vertical direction; a distortion coefficient acquisition module 64, configured to construct an fitness function based on the sum of squares of errors, and calculate the global optimal solution of the distortion coefficient according to the fitness function; a focal length acquisition module 66, configured to extract parallel line identification information in the current scene, and obtain the focal length closed solution based on the relative position relationship of the parallel lines.

[0116] Figure 6The camera intrinsic parameter calibration device shown directly utilizes linear structures (such as road markings, building edges, etc.) in the collected image in the current scene for distortion calibration, effectively solving the problem of dependence on a calibration board in the traditional method. The calibration process is simplified, and the camera intrinsic parameter calibration is completed only by relying on the linear features in the road scene, so that the camera can be adaptively calibrated in a dynamic scene, effectively improving the accuracy and stability of the calibration.

[0117] It should be noted that the camera intrinsic parameter calibration device described above can correspond to the Figures 1 to 5 The corresponding related description and effects in the embodiments shown are not repeated here.

[0118] According to the embodiments of the present application, an electronic device is also provided.

[0119] Figure 7 is a structural block diagram of an electronic device according to the preferred embodiments of the present application. As shown in Figure 7 The electronic device according to the present application includes a memory 70 and a processor 72, the memory 70 is used to store the computer execution instructions of the processor 70, and the processor 72 is configured to execute any of the camera intrinsic parameter calibration methods described above by executing the executable instructions.

[0120] The processor 72 can be a central processing unit (CPU). The processor 72 can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, or combinations thereof.

[0121] The memory 70 is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as the program instructions / modules corresponding to the camera intrinsic parameter calibration method in the embodiments of the present application. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory.

[0122] The memory 70 can include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required by a function, and the data storage area can store data created by the processor and the like. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory 70 can optionally include a memory disposed remotely with respect to the processor, which can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0123] The one or more modules described above are stored in the memory 70 described above, and when executed by the processor 72, perform the camera intrinsic calibration method as described in the embodiments shown in the drawings. Figures 1 to 5 The camera intrinsic calibration method in the embodiments shown in the drawings.

[0124] The specific details of the electronic device described above can be understood by referring to the corresponding related descriptions and effects of the embodiments shown in the drawings, which will not be described here. Figures 1 to 5 The specific details of the electronic device described above can be understood by referring to the corresponding related descriptions and effects of the embodiments shown in the drawings, which will not be described here.

[0125] In summary, by means of the above embodiments provided by the present application, the camera intrinsic calibration process is realized by utilizing the linear structures (such as road markings, building edges, etc.) existing in the images collected in the current scene, effectively solving the problem of dependence on calibration boards in related methods. By accurately extracting the boundary pixel points of the curved curve in the image through the Mobile-Seed network, the distortion center is located using Gaussian curve fitting, and the improved genetic algorithm is used to optimize the distortion coefficient. In addition, the spatial parallel line constraint is introduced, and the focal length of the camera is calculated by optimization, further improving the intrinsic calibration. This method simplifies the calibration process and only relies on the linear features in the road scene to complete the camera intrinsic estimation, so that the camera can be adaptively calibrated in a dynamic scene, effectively improving the accuracy and stability of the calibration.

[0126] The above detailed description further describes the purpose, technical solutions and beneficial effects of the present application, and it should be understood that the above is only a specific embodiment of the present application and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solutions of the present application shall be included in the protection scope of the present application.

Claims

1. A camera intrinsic parameter calibration method, characterized in that, The method comprises the following steps: Collecting an image in a current scene, identifying boundary pixel points of a linear structure with distortion in the collected image, and obtaining boundary position information of the linear structure; Sampling the boundary of the linear structure in the horizontal direction and the vertical direction according to the boundary position information, and fitting the sampling points to obtain the mean position of the boundary as the distortion center in the horizontal direction and the vertical direction; Constructing a fitness function based on the sum of squares of errors, and calculating a global optimal solution of distortion coefficients according to the fitness function; Extracting parallel line identification information in the current scene, and obtaining a focal length closed solution based on the relative position relationship of the parallel lines.

2. The method of claim 1, wherein, The method comprises the following steps: Using a semantic segmentation structure of a Mobile-Seed network to identify a region of the linear structure with distortion in the collected image; Using a boundary detection structure of the Mobile-Seed network to extract boundary pixel points of the linear structure with distortion in the collected image, and obtaining boundary position information of the linear structure.

3. The method of claim 1, wherein, The method comprises the following steps: Sampling the boundary of the linear structure in the horizontal direction and the vertical direction, and obtaining a plurality of discrete sampling points as input data for Gaussian curve fitting; Obtaining the mean position of the distorted curve of the linear structure by performing Gaussian curve fitting on the sampling points, as the distortion center in the horizontal direction and the vertical direction.

4. The method of claim 1, wherein, The method comprises the following steps: Constructing the fitness function Fitness by the following formula: wherein, is the number of sampled distortion-corrected image points, is the i-th distortion-corrected de-distorted point coordinate, is the straight line obtained by least square fitting of the de-distorted point coordinates, is the slope of the fitted straight line, is the intercept of the fitted straight line; In the case that the value of the fitness function Fitness is minimum, the radial distortion coefficients c1, c2 and the tangential distortion coefficients d1, d2 are taken as the global optimal solution of the distortion coefficients.

5. The method of claim 4, wherein, The method comprises the following steps: S1: applying randomly initialized radial distortion coefficients c1, c2 and tangential distortion coefficients d1, d2 to the sampling points obtained by sampling the boundary of the linear structure with distortion in the image for distortion correction; S2: obtaining the straight line by least square fitting the coordinates of the sampling points after distortion correction ; S3: substituting the coordinates of the n sampling points after distortion correction, the values of m and b into the fitness function formula to calculate the value of the fitness function; S4: cyclically executing S1 to S3, dynamically adjusting c1, c2, d1, d2 each time, and searching iteratively to obtain the corresponding radial distortion coefficients c1, c2 and tangential distortion coefficients d1, d2 as the global optimal solution of the distortion coefficients in the case that the value of the fitness function Fitness is minimum.

6. The method of claim 1, wherein, The focal length closed solution is obtained based on the relative position relationship of the parallel lines in the current scene, and the focal length closed solution comprises: In the current scene, a group or multiple groups of parallel lines are extracted according to specific marks; For each group of parallel lines, a unique solution of a projection depth corresponding to multiple end points of the group of parallel lines is obtained according to image point coordinates corresponding to the multiple end points; A focal length closed solution corresponding to the group of parallel lines is obtained according to the unique solution of the projection depth, the distortion center, and the image point coordinates; The calculation results of the focal length closed solutions corresponding to the one or more groups of parallel lines are integrated to determine a final focal length closed solution.

7. The method of claim 6, wherein, For each group of parallel lines, a unique solution of a projection depth corresponding to multiple end points of the group of parallel lines is obtained according to image point coordinates corresponding to the multiple end points, and the unique solution of the projection depth comprises: A binocular stereo vision system is constructed, and an origin of a world coordinate system is set to be located at a light center of one of the binoculars; The coordinates of the multiple end points of the group of parallel lines in the world coordinate system and the image point coordinates corresponding to the multiple end points are substituted into a projection equation; An association between the projection depth and the image point coordinates is found according to the projection equation to obtain a constraint equation; The projection depth is set to be positive, and all effective three-dimensional points in the image are located in front of the camera, and in the case that the image point coordinates are known, a zero space of a matrix composed of the image point coordinates is solved to obtain a unique solution of the projection depth corresponding to the multiple end points.

8. The method of claim 6, wherein, A focal length closed solution corresponding to the group of parallel lines is obtained according to the unique solution of the projection depth, the distortion center position, and the image point coordinates, and the focal length closed solution f is obtained through the following formula: The focal length closed solution f is obtained through the following formula: where , and are the i-th elements of the vectors and , , , is the center of distortion, , are the epipolar depths of the two end points of a line segment in the set of parallel lines, respectively, are the epipolar depths of the two end points of another line segment in the set of parallel lines, which is equal in length to the line segment, respectively, are the image point coordinates corresponding to the two end points of the line segment, and are the image point coordinates corresponding to the two end points of the another line segment.

9. A camera intrinsic parameter calibration device, characterized in that, The recognition module is configured to collect an image in a current scene, recognize boundary pixel points of a linear structure with distortion in the collected image, and obtain boundary position information of the linear structure; The distortion center obtaining module is configured to sample boundaries of the linear structure in a horizontal direction and a vertical direction according to the boundary position information, and fit the sampling points to obtain a mean position of the boundaries as the distortion center positions in the horizontal direction and the vertical direction; The distortion coefficient obtaining module is configured to construct an fitness function based on a sum of squares of errors, and calculate a global optimal solution of a distortion coefficient according to the fitness function; The focal length obtaining module is configured to extract parallel line identification information in the current scene, and obtain a focal length closed solution based on a relative position relationship of the parallel lines. The processor and the memory are configured to execute the executable instructions of the camera intrinsic parameter calibration method according to any one of claims 1 to 8.

10. An electronic device, comprising: ​ ​

Images