Camera unit internal reference calibration method and related equipment

Abstract

The invention discloses a camera unit internal reference calibration method and related equipment, and relates to the field of camera unit parameter calibration, and the method comprises the steps: obtaining all potential checkerboard corner information; performing connected domain check on all the checkerboard angular point information to obtain a plurality of connected domains; performing screening operation based on the size of the external rectangle of each connected domain to obtain a plurality of effective connected domains; performing feature value resetting operation according to the range of each effective connected domain to obtain a reset feature value image; performing non-maximum suppression operation on the reset feature value image to obtain a suppressed feature value image; determining an initial center point based on the suppressed feature value image; determining a critical point based on the starting center point to obtain an initial point array; performing expansion operation on the initial lattice array to obtain a final lattice array; and point position correspondence is carried out based on the final point array and the coordinates of the target plate, so that the internal reference calibration operation of the to-be-calibrated camera unit can be completed.

Description

Technical Field

[0001] The present invention relates to the field of camera unit parameter calibration, and more specifically, to a camera unit internal parameter calibration method and related equipment. Background Art

[0002] Camera unit intrinsic calibration is a basic and critical task in the field of computer vision, which aims to determine the intrinsic properties of the camera, such as focal length, optical center position, distortion coefficient, etc. The accuracy of intrinsic calibration directly affects the effect of subsequent visual algorithms, such as 3D reconstruction, target tracking, position positioning, etc.

[0003] Existing intrinsic calibration algorithms are usually completed by identifying feature points of array patterns, such as checkerboard corner points, dots, etc. And these feature point arrays must satisfy the complete array, such as (m, n) array, then the total number of feature points must be m*n. If the feature points are missing, intrinsic calibration cannot be performed. Therefore, it is necessary to propose a more adaptable camera unit intrinsic calibration method. Summary of the invention

[0004] A series of simplified concepts are introduced in the Summary of the Invention section, which will be further described in detail in the Detailed Description of the Invention section. The Summary of the Invention section of this application does not mean to attempt to define the key features and essential technical features of the claimed technical solution, nor does it mean to attempt to determine the scope of protection of the claimed technical solution.

[0005] In a first aspect, the present application proposes a method for calibrating an intrinsic parameter of a camera unit, comprising:

[0006] Get all potential checkerboard corner point information;

[0007] Perform a connected domain check on all chessboard corner point information to obtain multiple connected domains;

[0008] Perform a screening operation based on the size of the circumscribed rectangle of each of the above connected domains to obtain multiple valid connected domains;

[0009] Perform an eigenvalue reset operation according to the range of each of the above-mentioned valid connected domains to obtain a reset eigenvalue image;

[0010] Performing a non-maximum suppression operation on the reset eigenvalue image to obtain a suppressed eigenvalue image;

[0011] Determine the starting center point based on the above suppressed eigenvalue image;

[0012] Determine adjacent points based on the above starting center point to obtain an initial point array;

[0013] Performing an expansion operation on the above initial point array to obtain a final point array;

[0014] Point correspondence is performed based on the above-mentioned final point array and the target plate coordinates to complete the internal parameter calibration operation of the camera unit to be calibrated.

[0015] In a feasible implementation manner, before the above step of obtaining all potential checkerboard corner point information, the following step is also included:

[0016] Obtaining the original image of the target plate through the camera unit to be calibrated;

[0017] Obtain a checkerboard image based on the original image;

[0018] The checkerboard corner feature extraction is performed on the above checkerboard image to obtain all the potential checkerboard corner point information.

[0019] In a feasible implementation, the screening operation is performed based on the size of the circumscribed rectangle of each of the connected domains to obtain multiple valid connected domains, including:

[0020] A screening operation is performed based on the size of the circumscribed rectangle of each of the above connected domains, and connected domains whose length and width are both greater than the preset pixels are retained to determine multiple valid connected domains.

[0021] In a feasible implementation manner, the above-mentioned eigenvalue resetting operation is performed according to the range of each of the above-mentioned valid connected domains to obtain a reset eigenvalue image, including:

[0022] The feature points outside the effective connected domain are set to 0, and the original values ​​of the specific points within the effective connected domain are retained to obtain the reset feature value image.

[0023] In a feasible implementation manner, performing a non-maximum suppression operation on the reset eigenvalue image to obtain a suppressed eigenvalue image includes:

[0024] Selecting a preset multiple of the average value of the top N largest feature point values ​​in the reset feature value image as the initial threshold;

[0025] Optimize the upper and lower limits of the above initial threshold to obtain a reference threshold;

[0026] After the above reset, the eigenvalue image performs a traversal operation on each point based on a search box of a preset size;

[0027] When the current point is the maximum value point in the search box and the current point is greater than the above reference threshold, the original value of the current point is retained;

[0028] When other points are the maximum value points in the search box, the current point is set to 0;

[0029] The image after the traversal operation and the reset operation is used as the above-mentioned suppressed eigenvalue image.

[0030] In a feasible implementation manner, the above-mentioned determining adjacent points based on the above-mentioned starting center point to obtain the initial point array includes:

[0031] Determine adjacent points in four standard directions based on the screening candidate points and the block distances in the four standard directions according to the starting center point, wherein the four standard directions include up, down, left and right;

[0032] Inferring feature points in four reference directions based on the starting center point and the adjacent points in the four standard directions to determine the adjacent points in the four reference directions, wherein the four reference directions include upper left, upper right, lower left and lower right;

[0033] The starting center point, the adjacent points in the four standard directions and the adjacent points in the four reference directions are used as an initial point array.

[0034] In a feasible implementation manner, the above-mentioned expansion operation is performed on the above-mentioned initial point array to obtain the final point array, including:

[0035] Obtaining the horizontal arrangement standard deviation rate and the vertical arrangement standard deviation rate of the initial point array;

[0036] When both the horizontal arrangement standard deviation rate and the vertical arrangement standard deviation rate are less than the deviation rate threshold, continue to expand in the standard direction and the reference direction based on the potential position formula to obtain an expanded array until all the newly added points of the expanded array are virtual points and then stop expanding;

[0037] The array after expansion before stopping expansion is used as the final point array.

[0038] In a second aspect, the present application proposes a camera unit internal reference calibration device, comprising:

[0039] A first acquisition unit is used to acquire all potential checkerboard corner point information;

[0040] A second acquisition unit is used to perform a connected domain check on all the checkerboard corner point information to obtain multiple connected domains;

[0041] A third acquisition unit, configured to perform a screening operation based on the size of the circumscribed rectangle of each connected domain to acquire a plurality of valid connected domains;

[0042] A fourth acquisition unit, configured to perform a feature value resetting operation according to the range of each of the valid connected domains to obtain a reset feature value image;

[0043] A fifth acquisition unit, configured to perform a non-maximum suppression operation on the reset eigenvalue image to obtain a suppressed eigenvalue image;

[0044] A determination unit, configured to determine a starting center point based on the suppressed eigenvalue image;

[0045] A sixth acquisition unit, configured to determine adjacent points based on the starting center point to acquire an initial point array;

[0046] a seventh acquisition unit, configured to perform an expansion operation on the initial point array to acquire a final point array;

[0047] The calibration unit is used to perform point correspondence based on the final point array and the target plate coordinates to complete the internal parameter calibration operation of the camera unit to be calibrated.

[0048] In a third aspect, an electronic device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is used to implement the steps of the camera unit intrinsic parameter calibration method of any one of the above-mentioned first aspects when executing the computer program stored in the memory.

[0049] In a fourth aspect, the present application further proposes a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for calibrating the intrinsic parameters of the camera unit of any one of the first aspects is implemented.

[0050] In summary, traditional intrinsic calibration algorithms usually rely on a complete array of feature points, such as a (m, n) checkerboard. If the checkerboard in the image is incomplete or some corner points are not detected for various reasons (such as occlusion, insufficient illumination), these algorithms may not be executed or their calibration accuracy is greatly reduced. The present application flexibly identifies and utilizes the actual visible feature points in the image, and can perform effective intrinsic calibration even when the feature points are partially missing. This increases the scope of application and environmental adaptability of the algorithm, making it available in a wider range of practical application scenarios. In the fourteenth application, a connected domain analysis is performed after feature point detection, which not only helps to distinguish independent feature point groups, but also can filter out valid connected domains by the size and shape of the circumscribed rectangle. By resetting the image feature values ​​in the effective connected domain and performing non-maximum suppression, the detection accuracy of the corner points is further improved, ensuring that the corner points finally used for calibration are the real and strongest feature points in the image, reducing the false positive rate, and optimizing the quality of the corner points and the reliability of calibration. Compared with traditional methods, this scheme not only relies on the integrity of the initial point array, but can dynamically expand the point array based on the points actually detected. This expansion strategy enhances the ability to handle irregular or partially occluded checkerboards, allowing the algorithm to work effectively even under incomplete data conditions. Finally, based on the correspondence between the dynamically expanded point array and the coordinates of the target plate, this solution uses calibration algorithms such as Zhang Zhengyou to calculate the camera internal parameters, which ensures high-precision calibration results. The present application scheme provides a more flexible and robust camera unit internal parameter calibration method, which is particularly suitable for use in environments where feature points are incomplete or conditions vary greatly. By optimizing the feature point detection and processing flow, the scheme significantly improves the success rate and accuracy of calibration, and adapts to a wider range of application needs. This makes the method have important practical application value in the fields of industry, autonomous driving, robot vision, etc. The camera unit internal parameter calibration method proposed in this application, and other advantages, objectives and features of this application will be partially reflected in the following description, and partly will be understood by technicians in this field through research and practice of this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present specification. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0052] Figure 1 A schematic diagram of the process of calibrating the intrinsic parameters of a camera unit provided in an embodiment of the present application;

[0053] Figure 2 A schematic diagram of a checkerboard image extraction principle provided in an embodiment of the present application;

[0054] Figure 3 A schematic diagram of the principle of extracting chessboard corner point information provided in an embodiment of the present application;

[0055] Figure 4 A schematic diagram of a reset eigenvalue image provided in an embodiment of the present application;

[0056] Figure 5 A schematic diagram of a suppressed eigenvalue image provided in an embodiment of the present application;

[0057] Figure 6 A schematic diagram of adjacent points in four standard directions provided in an embodiment of the present application;

[0058] Figure 7 A schematic diagram of adjacent points in four reference directions provided in an embodiment of the present application;

[0059] Figure 8 A schematic diagram of an extended operation principle provided in an embodiment of the present application;

[0060] Fig. 9 A schematic diagram after an extended operation is provided in an embodiment of the present application;

[0061] Fig.10 Another schematic diagram after the expansion operation provided in the embodiment of the present application;

[0062] Fig.11 A schematic diagram of an internal reference calibration scenario provided in an embodiment of the present application;

[0063] Fig.12 A schematic diagram of the structure of an internal reference calibration device for an imaging unit provided in an embodiment of the present application;

[0064] Fig.13 A schematic diagram of the structure of an electronic device for calibrating the internal parameters of a camera unit provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0066] See also Figure 1 , is a schematic diagram of a process flow of a camera unit internal parameter calibration method provided in an embodiment of the present application, which may specifically include:

[0067] S110, obtaining all potential chessboard corner point information;

[0068] For example, a corner detection algorithm, such as the Harris corner detector or the Shi-Tomasi algorithm, can be used to identify potential corners in the image. These corners are where the vertical and horizontal lines in the checkerboard pattern meet.

[0069] S120, performing a connected domain check on all chessboard corner point information to obtain multiple connected domains;

[0070] For example, among the feature points of all chessboard corner point information, there are many potential interference points that need to be further optimized. Differentiate each independent chessboard set, analyze the spatial proximity of corner points, identify continuous adjacent corner point groups, and form multiple connected domains.

[0071] Specifically, we check the connected domain. For any point with an eigenvalue greater than 1.0, we check the 8 points in the circle around it. If the eigenvalue of a point in the vicinity is greater than the 0.3 threshold, then this point is added to the point set, and the points around these points are checked again until there are no points with a value greater than the 0.3 threshold. The collection area of ​​these points is a complete connected domain. After checking the entire image, we get many connected domains.

[0072] S130, performing a screening operation based on the size of the circumscribed rectangle of each of the above connected domains to obtain multiple valid connected domains;

[0073] Exemplarily, the bounding rectangle of each connected domain is calculated, and based on the size (and possible shape) of the rectangle, it is determined which connected domains may effectively represent the actual checkerboard region.

[0074] S140, performing an eigenvalue resetting operation according to the range of each of the above-mentioned valid connected domains to obtain a reset eigenvalue image;

[0075] Exemplarily, within the verified effective connected domain, the feature values ​​of the image are reset to highlight the corner features.

[0076] S150, performing a non-maximum suppression operation on the reset eigenvalue image to obtain a suppressed eigenvalue image;

[0077] Exemplarily, non-maximum suppression is applied to the eigenvalue image to remove high eigenvalues ​​in non-corner regions and retain the true strongest corners, where the strongest corners refer to the corners with the highest response value among the corners identified by the corner detection algorithm in the image. The high eigenvalues ​​in non-corner regions are suppressed, thereby retaining the true strongest corners to form a suppressed eigenvalue image.

[0078] S160, determining a starting center point based on the suppressed eigenvalue image;

[0079] Exemplarily, a starting point is selected as a reference for chessboard recognition, and a corner point at a central position is selected in the suppressed eigenvalue image as a starting point for subsequent point array recognition.

[0080] Specifically, use all the feature points left after non-maximum suppression to calculate the centroid position b, then calculate the center c of the circumscribed rectangle of these points, and finally determine the center position d = (b + c) / 2. Find the point e closest to the center position d among these feature points. This point is used as the starting center point.

[0081] S170, determining adjacent points based on the above starting center point to obtain an initial point array;

[0082] Exemplarily, starting from the selected starting center point, the directly adjacent corner points around it are determined to form an initial chessboard point array.

[0083] S180, performing an expansion operation on the initial point array to obtain a final point array;

[0084] Exemplarily, according to the layout and direction of the initial point array, more checkerboard corner points are searched outward to obtain a final point array, and the final point array is not necessarily a complete checkerboard array.

[0085] S190, performing point correspondence based on the final point array and the target plate coordinates to complete the internal reference calibration operation of the camera unit to be calibrated.

[0086] For example, Fig.11 As shown, it is a schematic diagram of an internal parameter calibration scenario provided by an embodiment of the present application. The final point array recognized in the image corresponds to the chessboard in the real world, that is, the coordinates of the target plate, and the internal parameters of the camera, such as focal length, optical center, etc., are calculated using algorithms such as Zhang Zhengyou calibration method.

[0087] It should be noted that the target plate refers to a physical plate with a known pattern (such as a checkerboard) used in the intrinsic calibration process of the camera unit. The target plate is usually printed with a regular pattern (such as a checkerboard, a dot array, etc.), and the geometric information of this pattern (including the relative position of the corner points) is known and fixed. During the intrinsic calibration process, the camera captures the image of the target plate, identifies the pattern feature points in the image (such as the corner points of the checkerboard), and matches these points in the image with the actual physical points on the plate, thereby calculating the camera's intrinsic parameters, such as focal length, optical center position, distortion coefficient, etc.

[0088] In summary, traditional intrinsic calibration algorithms usually rely on a complete array of feature points, such as a (m, n) chessboard. If the chessboard in the image is incomplete or some corner points are not detected due to various reasons (such as occlusion, insufficient illumination), these algorithms may not be executed or their calibration accuracy is greatly reduced. The present application flexibly identifies and utilizes the actual visible feature points in the image, and can perform effective intrinsic calibration even when the feature points are partially missing. This increases the scope of application and environmental adaptability of the algorithm, making it available in a wider range of practical application scenarios. The embodiment of the present application performs connected domain analysis after feature point detection, which not only helps to distinguish independent feature point groups, but also can filter out effective connected domains by the size and shape of the circumscribed rectangle. By resetting the image feature values ​​in the effective connected domain and performing non-maximum suppression, the detection accuracy of the corner points is further improved, ensuring that the corner points finally used for calibration are the real and strongest feature points in the image, reducing the false positive rate, and optimizing the quality of the corner points and the reliability of calibration. Compared with the traditional method, this solution not only relies on the integrity of the initial point array, but can dynamically expand the point array based on the points actually detected. This expansion strategy enhances the ability to handle irregular or partially occluded checkerboards, allowing the algorithm to work effectively even under incomplete data conditions. Finally, based on the correspondence between the dynamically expanded point array and the coordinates of the target plate, this solution uses calibration algorithms such as Zhang Zhengyou to calculate the camera intrinsic parameters, which ensures high-precision calibration results. The present application scheme provides a more flexible and robust method for calibrating the intrinsic parameters of a camera unit, which is particularly suitable for use in environments where feature points are incomplete or conditions vary greatly. By optimizing the feature point detection and processing process, the scheme significantly improves the success rate and accuracy of calibration, and adapts to a wider range of application needs. This makes the method have important practical application value in the fields of industry, autonomous driving, and robotic vision.

[0089] In some examples, before the step of obtaining all potential checkerboard corner point information, the following step is further included:

[0090] Obtaining the original image of the target plate through the camera unit to be calibrated;

[0091] Obtain a checkerboard image based on the original image;

[0092] The checkerboard corner feature extraction is performed on the above checkerboard image to obtain all the potential checkerboard corner point information.

[0093] For example, Figure 2 A schematic diagram of a checkerboard image extraction principle provided in an embodiment of the present application; Figure 3A schematic diagram of the principle of extracting checkerboard corner information provided in an embodiment of the present application. The original image of the target target plate is captured using the camera unit to be calibrated. The target plate usually contains a standard checkerboard pattern, and its geometric features (such as the number and size of the grids) are predefined and known. The original image provides a visual basis for identifying and calculating the intrinsic parameters of the camera.

[0094] Separate the checkerboard part from the original image. This involves image preprocessing operations such as cropping, rotation correction, and grayscale conversion to make the checkerboard more clearly identifiable.

[0095] Apply a corner detection algorithm such as Harris or Shi-Tomasi to identify all potential checkerboard corners. These corners are the intersections of checkerboard rows and columns and have high contrast features that make them easy to identify in the image. Corner points are key data points for camera calibration calculations. By accurately identifying these points, the accuracy of the calibration process can be ensured because the image coordinates of these points will be mapped to their corresponding world coordinates to calculate the camera's intrinsic parameters.

[0096] In some examples, the screening operation is performed based on the size of the bounding rectangle of each of the connected domains to obtain multiple valid connected domains, including:

[0097] A screening operation is performed based on the size of the circumscribed rectangle of each of the above connected domains, and connected domains whose length and width are both greater than the preset pixels are retained to determine multiple valid connected domains.

[0098] For example, in real images, many feature points that are not checkerboard corners may be identified due to noise, illumination changes, etc. These points may be randomly distributed or form small connected domains that do not conform to the characteristics of a checkerboard. Retaining only those large and regular connected domains can greatly improve the data quality and accuracy of subsequent processing steps, because these connected domains are more likely to be true checkerboard areas.

[0099] First, calculate an outer rectangle for each connected domain, that is, the smallest rectangular box that completely covers the connected domain. Set a preset pixel size, which is usually determined according to the actual chessboard size and image resolution. This threshold reflects the size of a connected domain.

[0100] The minimum size of a connected domain to be considered as a valid chessboard. Check whether the length and width of the bounding rectangle of each connected domain are greater than the preset pixel threshold. Only when the bounding rectangle of the connected domain satisfies the length and width greater than this threshold at the same time, the connected domain is retained as a valid connected domain.

[0101] For example: confirm the size of the bounding rectangle of the connected domain, and then only keep the connected domain whose length and width are both greater than 10 pixels.

[0102] In summary, through such a screening operation, larger connected domains can be effectively filtered out, which are usually complete or nearly complete checkerboard regions, and their structure and size conform to the characteristics of the checkerboard. This screening helps reduce the amount of calculation and complexity in subsequent steps, while improving the accuracy and reliability of the calibration process.

[0103] In some examples, performing the eigenvalue resetting operation according to the range of each of the valid connected domains to obtain a reset eigenvalue image includes:

[0104] The feature points outside the effective connected domain are set to 0, and the original values ​​of the specific points within the effective connected domain are retained to obtain the reset feature value image.

[0105] For example, Figure 4 A schematic diagram of a reset eigenvalue image provided for an embodiment of the present application. By resetting the eigenvalues, the algorithm can focus on processing and analyzing those image areas that are most likely to contain necessary information, remove or weaken irrelevant or interfering features in the image, and avoid introducing errors or uncertainties in subsequent processing steps. For each feature point determined to be within a valid connected domain, its eigenvalue is kept unchanged. For feature points that do not belong to any valid connected domain, their eigenvalues ​​are set to 0. These points will not be considered in subsequent processing, thereby reducing interference from background noise and irrelevant information.

[0106] After the above processing, the resulting image will only show corner features within the valid connected domain, and the other areas will be 0. Such an image is more effective for accurate corner location and feature analysis. This processing method provides a clearer and more focused feature map for subsequent corner detection, feature analysis, and final camera calibration, allowing the algorithm to execute more effectively. Through this feature value reset operation, the reliability and efficiency of the entire calibration process are improved, while also ensuring the accuracy of the calibration results.

[0107] In some examples, performing a non-maximum suppression operation on the reset eigenvalue image to obtain a suppressed eigenvalue image includes:

[0108] Selecting a preset multiple of the average value of the top N largest feature point values ​​in the reset feature value image as the initial threshold;

[0109] Optimize the upper and lower limits of the above initial threshold to obtain a reference threshold;

[0110] After the above reset, the eigenvalue image performs a traversal operation on each point based on a search box of a preset size;

[0111] When the current point is the maximum value point in the search box and the current point is greater than the above reference threshold, the original value of the current point is retained;

[0112] When other points are the maximum value points in the search box, the current point is set to 0;

[0113] The image after the traversal operation and the reset operation is used as the above-mentioned suppressed eigenvalue image.

[0114] For example, Figure 5 A schematic diagram of a suppressed eigenvalue image provided in an embodiment of the present application. The top N feature point values ​​in the reset eigenvalue image are selected and their average values ​​are calculated. This average value is multiplied by a preset multiple to determine the initial threshold. This multiple is usually set based on experiments or experience, with the aim of ensuring that the threshold is neither too high to exclude valid feature points nor too low to introduce noise. The initial threshold is optimized and a reasonable upper and lower limit is set. This step is to adapt to the specific conditions of the image, such as the distribution density and contrast of feature points, so that the threshold is more adapted to the characteristics of the current image. In the reset image, a search box of a preset size is set for each pixel point for traversal. If the current point is the maximum value point in the search box and is greater than the reference threshold, the original value of the point is retained and it is considered to be a valid corner point. If the above conditions are not met, or if it is the maximum value point but is not greater than the reference threshold, the point is set to 0 and it is considered not to be a valid corner point. After completing the above traversal and reset operations, the resulting image is regarded as a suppressed eigenvalue image, which will be used for subsequent corner point analysis and camera calibration calculations.

[0115] Specifically, continue to use non-maximum suppression for image processing. Take the largest 400 feature point values, take the average, and divide it by 8 as the threshold a. Optimize the upper and lower limits of threshold a. When it is less than 1, set it to 1, when it is greater than 6, set it to 6, and when it is between 1-6, it is the calculated original value. Traverse each point in the image to confirm whether it is the maximum value in the range of 11*11. If not, continue to traverse other points and set the eigenvalue of this position to 0. If it is the maximum value in this range and the eigenvalue is greater than the threshold a, then retain the eigenvalue of this position. Finally, the eigenvalue map after non-maximum suppression is obtained, and at this time there are very few interference points.

[0116] In summary, through non-maximum suppression, each retained feature point in the image is the most obvious point in its neighborhood, which greatly reduces errors and blurred corners. The processed image provides more accurate and reliable data input for the calculation of camera intrinsic parameters, thereby improving the accuracy and reliability of calibration.

[0117] In some examples, the step of determining adjacent points based on the starting center point to obtain an initial point array includes:

[0118] Determine adjacent points in four standard directions based on the starting center point and the screening candidate points and the street block distances in four standard directions, wherein the four standard directions include up, down, left and right;

[0119] Inferring feature points in four reference directions based on the starting center point and the adjacent points in the four standard directions to determine the adjacent points in the four reference directions, wherein the four reference directions include upper left, upper right, lower left and lower right;

[0120] The starting center point, the adjacent points in the four standard directions and the adjacent points in the four reference directions are used as an initial point array.

[0121] For example, Figure 6 A schematic diagram of adjacent points in four standard directions provided in an embodiment of the present application; Figure 7 A schematic diagram of adjacent points in four reference directions provided for an embodiment of the present application. The point with the strongest eigenvalue obtained by non-maximum suppression or the centroid of all feature points is calculated as the starting center point. Screening based on four standard directions: including four basic directions: up, down, left, and right. In each direction, starting from the starting center point, search for the point that is in the same straight line as the center point and closest to it as a candidate point. In this process, the block distance (i.e., Manhattan distance, which calculates the sum of the horizontal distance and the vertical distance between points) is used to evaluate the proximity of the candidate points, and the point with the smallest block distance is selected as the adjacent point in that direction.

[0122] Four reference directions are defined: including the upper left, upper right, lower left, and lower right diagonal directions. The adjacent points in the determined standard directions are used to infer the points in the four diagonal directions. For example, the upper left point can be inferred by combining the position information of the adjacent points on the left and above. The inference method may involve simple geometric calculations, such as using vector addition and subtraction or the midpoint formula. The inferred points need to be verified in the image to ensure that they actually exist in the image and maintain a reasonable geometric alignment with other known corner points.

[0123] Specifically, use the starting center point to find four adjacent points on the top, bottom, left and right sides. Take the top point as an example, and the principle of finding other points is similar. Figure 6 As shown, first select all the points above the center point, and then calculate the 5 points closest to the center point among these points. Then calculate the block distance, increase the horizontal distance difference by 5 times, and keep it unchanged in the vertical direction, add the two together, and then get the point closest to the center point in this block, which is the target point found. The principle is similar for other bottom, left and right positions.

[0124] Then, if Figure 7As shown, continue to find the four points of the upper left, upper right, lower left, and lower right. Because some points have been found, a more reliable method can be used to find these four points to reduce the impact of interference points. Take finding the upper left point as an example, and the principles of the other points are the same. Using the existing center point and the four points of the top, bottom, left, and right, the position of the upper left point can be inferred. The coordinate x of the upper left point = upper x-(center x-right x); coordinate y = right y-(center y-upper point y). In this way, we have the inferred position of the upper left point. Then, among all the feature points, find the point closest to the inferred position, and this distance is less than the threshold of 10 pixels, then it is the target upper left point we are looking for. The methods for the other upper right, lower left, and lower right points are similar.

[0125] In summary, the method based on accurate adjacent point determination provides a structured and highly ordered point array, which is essential for accurately calculating the intrinsic parameters of the camera. By ensuring the correctness and completeness of the initial point array, the accuracy and reliability of subsequent calibration operations can be effectively improved. In addition, this method also helps to improve the adaptability of the algorithm to complex image environments, including those where feature points are unevenly distributed or partially occluded.

[0126] In some examples, the above-mentioned expansion operation on the above-mentioned initial point array to obtain the final point array includes:

[0127] Obtaining the horizontal arrangement standard deviation rate and the vertical arrangement standard deviation rate of the initial point array;

[0128] When both the horizontal arrangement standard deviation rate and the vertical arrangement standard deviation rate are less than the deviation rate threshold, continue to expand in the standard direction and the reference direction based on the potential position formula to obtain an expanded array until all the newly added points of the expanded array are virtual points and then stop expanding;

[0129] The array after expansion before stopping expansion is used as the final point array.

[0130] For example, Figure 8 A schematic diagram of an extended operation principle provided in an embodiment of the present application; Fig. 9 A schematic diagram after an extended operation is provided in an embodiment of the present application; Fig.10 Another schematic diagram after the expansion operation provided by the embodiment of the present application. The standard deviation rate is a statistical measure used to evaluate the dispersion of a set of values ​​relative to its mean value, which measures the consistency of the horizontal and vertical arrangement of the points in the point array.

[0131] Horizontal standard deviation: Calculates the deviation of the horizontal position of each point in the point array from the average position of the entire row.

[0132] Longitudinal standard deviation: calculates the deviation of the vertical position of each point in the point array from the average position of the entire column.

[0133] A threshold is set, and the expansion will continue only when the standard deviation rate of the horizontal and vertical arrangement of the point array is less than this threshold. This threshold ensures that the expanded point array maintains sufficient regularity and consistency. If the standard deviation rate of both horizontal and vertical distributions meets the requirements, the expansion operation continues. This ensures the uniform distribution and symmetry of the point array in space, which helps to improve the accuracy of subsequent calculations.

[0134] The potential location of a new point is calculated based on an existing array of points using a preset formula. These formulas are usually based on the geometric relationship of the known points, such as equally spaced arrays or linear interpolation.

[0135] The extension includes the original standard directions (up, down, left, right) and the reference directions (upper left, upper right, lower left, lower right).

[0136] If the newly added points do not actually correspond to image feature points (i.e., virtual points), and all the newly added points during the expansion process are virtual points, the expansion stops. Once all the newly added points are virtual points during a certain expansion process, the expansion operation stops. The point array obtained after the last valid expansion before the expansion is stopped becomes the final point array.

[0137] Specifically, take the initial point array of 9 points as an example: check whether these 9 points meet the array arrangement. From the horizontal point of view, record the distance between the adjacent points on the left and right, calculate the standard deviation and the average value, and calculate the standard deviation rate = standard deviation / average value. If this value is greater than the threshold, then it is considered that it is not an array arrangement, and the search for these 9 points has failed; similarly, analyze the standard deviation rate from the vertical direction. If both meet the conditions at the same time and are less than this threshold, then these 9 points meet the conditions and are the target points to be found.

[0138] The 9 points in the center must exist. An incomplete chessboard usually has incomplete edges. If the 9 points in the center cannot be found completely, the search for the chessboard corners will fail. In other words, the smallest complete chessboard array is 3*3.

[0139] Next, we need to expand these 9 points from the four directions of up, down, left, and right. Take the upward direction as an example, and the principle of other directions is the same.

[0140] For example, if the points found so far are 3*3, then there are 3 points to look for upwards; if there are 4 points found horizontally, then there are also 4 points to look for upwards. Taking 3 points as an example, first calculate the position of the potential point. The potential position formula of the point = (x, y) = (p1.x*3-p2.x*3+p3.x, p1.y*3-p2.y*3+p3.y)

[0141] Using the potential position, use the same method as before to find the nearest point from all feature points. If it meets the threshold, the target point is found. If the target point is not found, then use the potential point position as a virtual point for subsequent continuous expansion. For each virtual point, the virtual value = 1 + the virtual value accumulation of the 3 points used. The virtual value of the target point actually found = 0;

[0142] After all the points in a row are aligned, the array mismatch of all points should be updated again. Formula = pixel distance between the inferred point and the actual point. And for the virtual point, find the point with the smallest virtual value and the smallest mismatch in each direction, and use it as the new coordinate to update the original virtual point.

[0143] After all points in this direction are refreshed, check whether there is a point with virtual value = 0 in the newly added row of points. If not, it means that all points are virtual points, and the search for this row fails. Otherwise, it is successful, that is, as long as there is a real point added, it is successful.

[0144] The expansion in one direction is completed. Similarly, the expansion in the four directions of up, down, left, and right is completed in sequence. The next round of expansion is repeated until no expansion in the four directions is successful in one cycle.

[0145] In summary, the point array expanded by this method can maximize the coverage of the relevant area of ​​the target image while ensuring consistency and regularity, and it is not necessary to capture the image of the entire chessboard during calibration. This expansion strategy not only improves the quality of the point array, but also optimizes the accuracy of subsequent intrinsic calibration calculations.

[0146] like Fig.12 As shown, the present application proposes a camera unit internal reference calibration device, comprising:

[0147] A first acquisition unit 21 is used to acquire all potential checkerboard corner point information;

[0148] A second acquisition unit 22 is used to perform a connected domain check on all the checkerboard corner point information to obtain multiple connected domains;

[0149] A third acquisition unit 23 is used to perform a screening operation based on the size of the circumscribed rectangle of each connected domain to obtain multiple valid connected domains;

[0150] A fourth acquisition unit 24 is used to perform a feature value resetting operation according to the range of each of the valid connected domains to obtain a reset feature value image;

[0151] A fifth acquisition unit 25, configured to perform a non-maximum suppression operation on the reset eigenvalue image to obtain a suppressed eigenvalue image;

[0152] A determination unit 26, configured to determine a starting center point based on the suppressed eigenvalue image;

[0153] A sixth acquisition unit 27, configured to determine adjacent points based on the starting center point to acquire an initial point array;

[0154] A seventh acquisition unit 28, configured to perform an expansion operation on the initial point array to obtain a final point array;

[0155] The calibration unit 29 is used to perform point correspondence based on the final point array and the target plate coordinates to complete the internal parameter calibration operation of the camera unit to be calibrated.

[0156] like Fig.13 As shown, an embodiment of the present application also provides an electronic device 300, including a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor, and when the processor 320 executes the computer program 311, the steps of any method of the above-mentioned camera unit internal parameter calibration are implemented.

[0157] Since the electronic device introduced in this embodiment is a device used to implement a camera unit internal parameter calibration device in the embodiment of the present application, based on the method introduced in the embodiment of the present application, technical personnel in this field can understand the specific implementation mode of the electronic device of this embodiment and its various variations. Therefore, how the electronic device implements the method in the embodiment of the present application is not introduced in detail here. As long as the equipment used by technical personnel in this field to implement the method in the embodiment of the present application is within the scope of protection of this application.

[0158] In the specific implementation process, when the computer program 311 is executed by the processor, it can achieve Figure 1 Any implementation manner in the corresponding embodiments.

[0159] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0160] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0161] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0162] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0163] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0164] An embodiment of the present application further provides a computer program product, which includes computer software instructions. When the computer software instructions are executed on a processing device, the processing device executes the process of camera unit internal parameter calibration in the corresponding embodiment.

[0165] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)), etc.

[0166] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0167] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0168] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0169] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0170] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0171] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for calibrating an internal parameter of a camera unit, characterized in that: include: Get all potential checkerboard corner point information; Perform a connected domain check on all chessboard corner point information to obtain multiple connected domains; Performing a screening operation based on the size of the circumscribed rectangle of each of the connected domains to obtain a plurality of valid connected domains; Performing an eigenvalue resetting operation according to the range of each of the valid connected domains to obtain a reset eigenvalue image; Performing a non-maximum suppression operation on the reset eigenvalue image to obtain a suppressed eigenvalue image; Determine a starting center point based on the suppressed eigenvalue image; Determine adjacent points based on the starting center point to obtain an initial point array; Performing an expansion operation on the initial point array to obtain a final point array; Point correspondence is performed based on the final point array and the target plate coordinates to complete the internal parameter calibration operation of the camera unit to be calibrated.

2. The camera unit internal parameter calibration method according to claim 1, characterized in that: Before the step of obtaining all potential checkerboard corner point information, the method further includes: Acquire an original image of the target plate through the camera unit to be calibrated; Acquire a checkerboard image based on the original image; Performing checkerboard corner point feature extraction on the checkerboard image to obtain all potential checkerboard corner point information.

3. The camera unit internal parameter calibration method according to claim 1, characterized in that: The screening operation is performed based on the size of the circumscribed rectangle of each connected domain to obtain multiple valid connected domains, including: A screening operation is performed based on the size of the circumscribed rectangle of each connected domain, and connected domains whose length and width are both greater than preset pixels are retained to determine a plurality of valid connected domains.

4. The camera unit internal parameter calibration method according to claim 1, characterized in that: The performing of an eigenvalue resetting operation according to the range of each of the valid connected domains to obtain a reset eigenvalue image includes: The feature points outside the effective connected domain are set to 0, and the original values ​​of the specific points within the effective connected domain are retained to obtain the reset feature value image.

5. The camera unit internal parameter calibration method according to claim 1, characterized in that: The performing a non-maximum suppression operation on the reset eigenvalue image to obtain a suppressed eigenvalue image includes: Selecting a preset multiple of the average value of the top N largest feature point values ​​in the reset feature value image as an initial threshold; Optimizing the upper and lower limits of the initial threshold to obtain a reference threshold; After the reset, the eigenvalue image performs a traversal operation on each point based on a search box of a preset size; When the current point is the maximum value point in the search box and the current point is greater than the reference threshold, retain the original value of the current point; When other points are the maximum value points in the search box, the current point is set to 0; The image after the traversal operation and the reset operation is used as the suppressed eigenvalue image.

6. The camera unit internal parameter calibration method according to claim 1, characterized in that: The step of determining adjacent points based on the starting center point to obtain an initial point array includes: Determine adjacent points in four standard directions based on the starting center point and the screening candidate points and the block distances in four standard directions, wherein the four standard directions include up, down, left and right; Perform feature point inference in four reference directions based on the starting center point and the adjacent points in the four standard directions to determine the adjacent points in the four reference directions, wherein the four reference directions include upper left, upper right, lower left and lower right; The starting center point, the adjacent points in the four standard directions and the adjacent points in the four reference directions are used as an initial point array.

7. The camera unit internal parameter calibration method according to claim 1, characterized in that: The step of performing an expansion operation on the initial point array to obtain a final point array includes: Obtaining a horizontal arrangement standard deviation rate and a vertical arrangement standard deviation rate of the initial point array; When both the horizontal arrangement standard deviation rate and the vertical arrangement standard deviation rate are less than the deviation rate threshold, continue to expand in the standard direction and the reference direction based on the potential position formula to obtain an expanded array until all the newly added points of the expanded array are virtual points and stop expanding; The array after expansion before stopping expansion is used as the final point array.

8. A camera unit internal parameter calibration device, characterized in that: include: A first acquisition unit is used to acquire all potential checkerboard corner point information; A second acquisition unit is used to perform a connected domain check on all the checkerboard corner point information to obtain multiple connected domains; A third acquisition unit, configured to perform a screening operation based on the size of the circumscribed rectangle of each connected domain to acquire a plurality of valid connected domains; A fourth acquisition unit, configured to perform a feature value resetting operation according to the range of each of the valid connected domains to obtain a reset feature value image; A fifth acquisition unit, configured to perform a non-maximum suppression operation on the reset eigenvalue image to obtain a suppressed eigenvalue image; A determination unit, configured to determine a starting center point based on the suppressed eigenvalue image; A sixth acquisition unit, configured to determine adjacent points based on the starting center point to acquire an initial point array; a seventh acquisition unit, configured to perform an expansion operation on the initial point array to acquire a final point array; The calibration unit is used to perform point correspondence based on the final point array and the target plate coordinates to complete the internal parameter calibration operation of the camera unit to be calibrated.

9. An electronic device, comprising: A memory and a processor, wherein the processor is used to implement the steps of the camera unit intrinsic parameter calibration method as described in any one of claims 1 to 7 when executing the computer program stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the camera unit intrinsic parameter calibration method as described in any one of claims 1 to 7 are implemented.

Images