Line array camera calibration method, device and equipment for scanning feature points and storage medium

Abstract

The application provides a line array camera calibration method, device and equipment for scanning feature points and a storage medium. By obtaining displacement information corresponding to different positions of a mobile platform and image data of a scanning object under multiple perspectives, homonymous mark points corresponding to the same physical structure under different perspectives are identified, a camera calibration model is established based on the displacement information and the image coordinates of the homonymous mark points, and joint solving of camera extrinsic parameters, camera intrinsic parameters and mark point space structure parameters is realized. Further, by constructing a re-projection model and optimizing and updating the calibration parameters using re-projection errors, high-precision camera calibration results are obtained. Compared with a traditional calibration method relying on regular calibration objects such as checkerboards, the application can directly use the feature structure of the scanning object itself to complete calibration, without the need for additional calibration objects, thereby effectively avoiding the influence of calibration object manufacturing errors and installation errors on calibration accuracy.

Description

Technical Field

[0001] This invention relates to the field of high-precision scanning measurement technology, and in particular to a method, apparatus, device, and storage medium for calibrating a linear array camera that scans feature points. Background Technology

[0002] Linear scan cameras, due to their high resolution and high imaging efficiency, are widely used in semiconductor wafer inspection, precision optical measurement, industrial vision inspection, and microscopic imaging. To ensure the accuracy of measurement results, line scan cameras typically require calibration to obtain parameters such as intrinsic and extrinsic parameters, as well as lens distortion. Current camera calibration methods mostly rely on calibration objects with regular geometric structures, such as checkerboard patterns or dot matrix boards. By detecting corner points or feature points on the calibration object, a correspondence between image coordinates and world coordinates is established, thereby solving for the camera parameters.

[0003] However, in high-precision scanning imaging scenarios, especially in detection applications with nanometer or submicron resolution, the small field of view per frame often means that the image can only contain a limited number of checkerboard corner points or regular features, making traditional checkerboard-based calibration methods difficult to implement effectively. Furthermore, the manufacturing precision, installation precision, and fixing method of the checkerboard and other calibration materials themselves introduce additional errors. When the manufacturing precision of the calibration material is lower than that of the object being inspected, the calibration material itself becomes a significant factor limiting calibration accuracy. In addition, under ultra-high resolution imaging conditions, diffraction effects, optical aberrations, and corner point positioning errors further affect the accuracy of feature point detection, thereby reducing the stability and reliability of the calibration results.

[0004] In summary, the problems existing in the current technology urgently need to be solved. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and storage medium for calibrating a linear scan camera that scans feature points, in order to overcome the deficiencies in the prior art and achieve calibration by directly utilizing the feature structure of the scanned object itself.

[0006] This invention provides a method for calibrating a linear array camera that scans feature points, comprising: The displacement information corresponding to the multiple movements of the mobile platform in the plane is obtained, and an image frame is captured after each movement. The image contains multiple marker points on the scanned object. Within the overlapping area of ​​adjacent frame images, identify the same-named marker points corresponding to the same physical structure. The same-named marker points are marker points that correspond to the same physical location from different viewpoints. A camera calibration model is established based on the displacement information and the image coordinates of the corresponding marker points; Based on the camera calibration model, the camera extrinsic parameters, camera intrinsic parameters, and marker point spatial structure parameters are jointly solved to obtain the initial camera calibration parameters. A reprojection model is constructed based on the camera calibration parameters, and the camera calibration parameters are optimized and updated according to the reprojection error to obtain the target camera calibration parameters.

[0007] According to the linear scan camera calibration method for scanning feature points provided by the present invention, the step of acquiring displacement information corresponding to multiple movements of the moving platform in the plane and acquiring an image frame after each movement specifically includes: The mobile platform is controlled to make multiple position adjustments in a two-dimensional plane according to a preset motion trajectory; After each position adjustment, the displacement coordinates of the mobile platform relative to the initial position are obtained as the displacement information corresponding to the current viewpoint; and the line scan camera is controlled to acquire images of the scanned object to obtain the image corresponding to the current viewpoint. The position adjustment, displacement acquisition, and image acquisition process is repeated to obtain displacement information and images corresponding to multiple different viewpoints; The images corresponding to two adjacent position adjustments have overlapping areas, so that at least some of the markers on the scanned object can be repeatedly observed from multiple perspectives.

[0008] According to the linear array camera calibration method for scanning feature points provided by the present invention, the displacement distance corresponding to two adjacent position adjustments is less than half of the field of view of the current view image.

[0009] According to the linear scan camera calibration method for scanning feature points provided by the present invention, the step of identifying the same-named marker points corresponding to the same physical structure within the overlapping area of ​​adjacent frame images specifically includes: In the first frame image, multiple target image regions with preset pattern features are identified, and the center position of each target image region is determined as a candidate marker point; Based on the displacement information corresponding to the mobile platform, the predicted position of each candidate marker point in the second frame image is determined; Image matching is performed near the predicted location to determine the matching image region corresponding to each target image region; The center position of the matched image region is determined as the corresponding marker point in the second frame image; Based on the matching relationship between candidate markers in the first frame image and corresponding markers in the second frame image, the same-name markers corresponding to the same physical structure are determined.

[0010] According to the linear array camera calibration method for scanning feature points provided by the present invention, the step of jointly solving the camera extrinsic parameters, camera intrinsic parameters, and marker point spatial structure parameters based on the camera calibration model to obtain initial camera calibration parameters specifically includes: Obtain the initial intrinsic parameters of the camera, and normalize the image coordinates of the corresponding marker points based on the initial intrinsic parameters; The first solution model is constructed based on the normalized image coordinates, the displacement information of the moving platform, and the camera height information. The camera extrinsic parameters and the spatial structure parameters of the marker points are obtained through linear solution. A second solution model is constructed based on the camera extrinsic parameters and the spatial structure parameters of the marker points, and the camera intrinsic parameters are obtained through linear solution. The obtained camera extrinsic parameters, camera intrinsic parameters, and marker point spatial structure parameters are used as the initial camera calibration parameters.

[0011] According to the linear array camera calibration method for scanning feature points provided by the present invention, the step of constructing a reprojection model based on the camera calibration parameters and optimizing and updating the camera calibration parameters according to the reprojection error to obtain the target camera calibration parameters specifically includes: A reprojection model between the spatial coordinates of the marker points and the image coordinates is established based on the initial camera calibration parameters. The predicted image coordinates of each corresponding marker point are calculated using the reprojection model. The reprojection error is determined based on the difference between the predicted image coordinates and the corresponding actual image coordinates; The initial camera calibration parameters are iteratively optimized with the goal of minimizing the reprojection error. When the reprojection error meets the preset convergence condition, the optimized target camera calibration parameters are output.

[0012] According to the linear array camera calibration method for scanning feature points provided by the present invention, the step of iteratively optimizing the initial camera calibration parameters with the goal of minimizing the reprojection error specifically includes: The camera intrinsic parameters, the camera extrinsic parameters, the lens distortion parameters, and the marker point spatial structure parameters are used as variables to be optimized. Construct a nonlinear optimization model with reprojection error as the objective function; The nonlinear least squares optimization algorithm is used to jointly solve the variables to be optimized in order to obtain the target camera calibration parameters.

[0013] The present invention also provides a calibration device for a linear scan camera that scans feature points, comprising: The information acquisition module is used to acquire displacement information corresponding to the multiple movements of the mobile platform in the plane, and to acquire an image frame after each movement, the image containing multiple marker points on the scanned object; The marker recognition module is used to identify the same-name marker points corresponding to the same physical structure in the overlapping area of ​​adjacent frame images. The same-name marker points are marker points that correspond to the same physical location from different viewpoints. The model building module is used to build a camera calibration model based on the displacement information and the image coordinates of the corresponding marker points; The parameter solving module is used to jointly solve the camera extrinsic parameters, camera intrinsic parameters, and marker point spatial structure parameters according to the camera calibration model to obtain the initial camera calibration parameters. The optimization and update module is used to construct a reprojection model based on the camera calibration parameters, and optimize and update the camera calibration parameters according to the reprojection error to obtain the target camera calibration parameters.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a linear scan camera calibration method for scanning feature points as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the linear scan camera calibration method for scanning feature points as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements a linear scan camera calibration method for scanning feature points as described above.

[0017] The present invention provides a linear array camera calibration method, apparatus, device, and storage medium for scanning feature points. By acquiring displacement information of a mobile platform at multiple positions and image data of the scanned object from different viewpoints, it identifies corresponding marker points with the same physical structure within the overlapping area of ​​adjacent frame images. Based on the displacement information and the image coordinates of the corresponding marker points, a camera calibration model is established, thereby achieving the joint solution of camera extrinsic parameters, camera intrinsic parameters, and marker point spatial structure parameters. Compared with traditional methods relying on checkerboard or regular dot matrix calibration objects, the present invention eliminates the need for additional calibration objects, directly utilizing the characteristic structure of the scanned object itself to complete the calibration process. This effectively avoids the adverse effects of manufacturing errors, installation errors, and fixing errors of calibration objects on the calibration results, thus improving the applicability of the method.

[0018] Furthermore, this invention establishes associations between corresponding marker points from multiple perspectives and constructs a camera calibration model by combining the displacement information of the mobile platform. This allows the calibration process to utilize not only local information from a single frame image but also the spatial constraints between multiple frames for joint solution. Compared to calibration methods that rely solely on single-frame feature relationships, this method can fully utilize redundant information in multi-view observation data, improving the stability and reliability of parameter estimation.

[0019] Furthermore, after obtaining the initial camera calibration parameters, the present invention constructs a reprojection model and uses the reprojection error to optimize and update the camera calibration parameters, so that the calibration parameters can continuously approach the real imaging model, thereby effectively reducing the reprojection error, improving the accuracy of camera parameter estimation, and obtaining more accurate calibration results.

[0020] Furthermore, the present invention adopts a calibration process of "calibration model construction - joint parameter solution - reprojection optimization", which can simultaneously take into account the solution efficiency and calibration accuracy. It is suitable for semiconductor wafer inspection, nanostructure measurement, high-resolution microscopic imaging and other application scenarios that require high-precision scanning measurement, and has good engineering application value and promotion prospects. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the linear array camera calibration method for scanning feature points provided by the present invention; Figure 2 This is a schematic diagram of the imaging system provided by the present invention; Figure 3 This is a schematic diagram of the structure of the linear array camera calibration device for scanning feature points provided by the present invention; Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] To address the problems in existing technologies, this invention proposes a linear scan camera calibration method for scanning feature points, enabling calibration to be completed directly using the feature structure of the scanned object itself. The following describes this linear scan camera calibration method for scanning feature points, as follows: Figure 1 As shown, including but not limited to the following steps: Step 110: Obtain the displacement information corresponding to the multiple movements of the mobile platform in the plane, and capture an image frame after each movement. The image contains multiple marker points on the scanned object.

[0025] In step 110, the moving platform causes the scanned object to change position within a two-dimensional plane, and the imaging system records the displacement information of the moving platform at each position. In the area array acquisition mode of the line scan camera, after each position movement of the moving platform, the line scan camera acquires image data corresponding to the current viewpoint. After multiple movements and image acquisitions, images from multiple different viewpoints and displacement information corresponding to each viewpoint are obtained. The scanned object contains marker points that can be repeatedly observed from multiple viewpoints. These marker points can originate from the scanned object's own identifiable structural features, texture features, or pattern features, and are used to establish the correspondence between different viewpoints.

[0026] Step 120: In the overlapping area of ​​adjacent frame images, identify the same-named marker points corresponding to the same physical structure. The same-named marker points are marker points that correspond to the same physical location from different viewpoints.

[0027] In step 120, the multi-frame images obtained in step 110 are analyzed and processed to determine the overlapping areas between adjacent viewpoints, and marker points corresponding to the same physical structure are found within the overlapping areas. For marker points that can appear repeatedly in multiple viewpoints and correspond to the same actual location, their cross-viewpoint correspondence is established, thereby obtaining a set of marker points with the same name. Through the set of marker points with the same name, image observation information from different viewpoints can be associated with a unified spatial structure, providing basic data for the subsequent establishment of a calibration model.

[0028] Step 130: Establish a camera calibration model based on the displacement information and the image coordinates of the corresponding marker points.

[0029] In step 130, the displacement information of the mobile platform at each viewpoint is used as motion constraint information, and the image coordinates of the corresponding marker points at each viewpoint are used as observation information to establish a camera calibration model describing the motion relationship of the mobile platform, the spatial position relationship of the marker points, and the camera imaging relationship. This camera calibration model is used to characterize the mapping relationship between camera parameters, the spatial structure of the marker points, and the image observation results, enabling joint analysis and solution of observation data obtained from different viewpoints within a unified model framework.

[0030] Specifically, for the camera extrinsic parameters, the coordinates in the base coordinate system are defined as follows: Image coordinates are ,in i Image sequence number j Given the calibration point number, the corresponding normalized image coordinates are:

[0031] in, f x , f y , c x , c y To estimate internal references, f x , f y Take the imaging system design values ​​as initial values. c x , c y Set the initial value to zero; D -1 This is a reverse distortion process.

[0032] Specify initial distortion coefficients k 1 , k 2 Then, for a certain image coordinate ( m, n The approximate value can be obtained from the following formula () m * , n * ):

[0033] in, The asymptotic symbol represents the magnitude of the remainder term in the asymptotic estimate.

[0034] These parameters are initially estimated by the user at the beginning of the algorithm and are solved iteratively in subsequent steps. For have:

[0035] in, i Image sequence number j The calibration point number; t c The camera height was obtained through measurement. x j ,y j() represents the coordinates of the calibration object coordinate system; To determine the rotation angle of the calibration object, This is the camera rotation angle.

[0036] Construct the following linear system to solve for the rotation angle of the calibration object. :

[0037] in, i Image sequence number j The calibration point number; t c The camera height was obtained through measurement; p j ，q j [Unknown world coordinates of the calibration point and rotation angle of the calibration object] Joint variables:

[0038] The above applies to all i , j Stacking them into a system of linear equations allows for the solution using least squares. , t c、 p j 、q j .

[0039] For the camera intrinsic parameters, first update them using the obtained extrinsic parameters:

[0040] Pick D j i =( m j i ) 2 +( n j i ) 2 Construct the following system of linear equations:

[0041] After stacking all the observation data, f x , f y , c x , c y Perform least squares fitting.

[0042] Step 140: Based on the camera calibration model, jointly solve the camera extrinsic parameters, camera intrinsic parameters, and marker point spatial structure parameters to obtain the initial camera calibration parameters.

[0043] In step 140, the parameters to be determined in the camera calibration model established in step 130 are solved. These parameters include extrinsic camera parameters reflecting camera attitude relationships, intrinsic camera parameters reflecting camera imaging characteristics, and spatial structure parameters describing the spatial distribution of marker points. By jointly solving these parameters, initial camera calibration parameters that describe the geometric and imaging relationships of the current imaging system are obtained, providing initial parameter values ​​for subsequent optimization processes.

[0044] Step 150: Construct a reprojection model based on the camera calibration parameters, and optimize and update the camera calibration parameters according to the reprojection error to obtain the target camera calibration parameters.

[0045] In step 150, a reprojection model is constructed based on the initial camera calibration parameters obtained in step 140. This model maps the spatial coordinates of the marker points to the image plane, obtaining the corresponding predicted image coordinates. Subsequently, the predicted image coordinates are compared with the actually observed image coordinates to obtain the reprojection error. The camera calibration parameters are iteratively optimized based on this reprojection error, gradually bringing the predicted results closer to the actual observations. Once the optimization process reaches a preset convergence condition, the optimized target camera calibration parameters are output. These target camera calibration parameters can more accurately reflect the imaging characteristics and geometric relationships of the actual imaging system, thereby improving the accuracy of subsequent measurement, detection, and image processing processes.

[0046] As a further optional embodiment, the step of obtaining the displacement information corresponding to the multiple movements of the mobile platform in the plane, and capturing an image frame after each movement, specifically includes: The mobile platform is controlled to make multiple position adjustments in a two-dimensional plane according to a preset motion trajectory; After each position adjustment, the displacement coordinates of the mobile platform relative to the initial position are obtained as the displacement information corresponding to the current viewpoint; and the line scan camera is controlled to acquire images of the scanned object to obtain the image corresponding to the current viewpoint. The position adjustment, displacement acquisition, and image acquisition process is repeated to obtain displacement information and images corresponding to multiple different viewpoints; The images corresponding to two adjacent position adjustments have overlapping areas, so that at least some of the markers on the scanned object can be repeatedly observed from multiple perspectives.

[0047] In this embodiment, the mobile platform serves as the carrier mechanism for the scanned object, driving the object to move within a two-dimensional plane to achieve repeated imaging of the same area from different perspectives. The preset motion trajectory can be set according to the actual application scenario, such as moving along the X direction, moving along the Y direction, moving along a combination of the X and Y directions, a serpentine scanning path, or other motion trajectories that can form multi-view observations. This embodiment does not limit this.

[0048] Specifically, after each position adjustment, the displacement coordinates of the mobile platform relative to its initial position are acquired, and these displacement coordinates are recorded as displacement information corresponding to the current viewpoint. Simultaneously, the linear scan camera is controlled to acquire images of the scanned object at the current position, obtaining image data from the corresponding viewpoint. By continuously executing the position adjustment, displacement recording, and image acquisition processes, image data from multiple viewpoints and their corresponding displacement information can be obtained, thereby forming a dataset for camera calibration.

[0049] Furthermore, to ensure the establishment of correspondences between different viewpoints, the images corresponding to two adjacent position adjustments have overlapping areas, allowing some marker points on the scanned object to appear simultaneously from multiple viewpoints. Since the same marker point can be repeatedly observed from multiple viewpoints, cross-viewpoint correlations can be established using the positional changes of the marker point in different images, providing basic data for subsequent identification of the same-name marker point and the construction of camera calibration models.

[0050] It should be noted that the displacement information can be obtained from the mobile platform's own position feedback device or from an external measuring device, as long as it reflects the relative position change of the mobile platform under different viewpoints. This embodiment does not impose specific limitations on this. By acquiring displacement information and image data corresponding to multiple viewpoints, a data foundation can be provided for subsequent camera calibration using the features of the scanned object itself, without relying on additional rule-based calibration objects.

[0051] As a further optional embodiment, the displacement distance corresponding to two adjacent position adjustments is less than half of the current viewpoint image field of view.

[0052] In this embodiment, to ensure a sufficient common observation area between adjacent viewpoints, the displacement distance corresponding to each position adjustment of the moving platform is controlled within half of the current viewpoint's field of view. By limiting the single movement distance, some marker points on the scanned object can appear simultaneously in two adjacent frames, thereby forming a stable overlapping observation area.

[0053] Since the subsequent calibration process requires establishing a correspondence using observations of the same marker point from multiple viewpoints, excessive movement distance may result in a lack of commonly observed marker points between adjacent viewpoints, thus affecting the accuracy of identifying marker points with the same name and the construction of subsequent calibration models. However, when the displacement distance is less than half the field of view of the current viewpoint, a high proportion of overlap between adjacent viewpoints can be ensured, allowing the same marker point to be repeatedly observed from multiple consecutive viewpoints, thereby improving the reliability of establishing cross-viewpoint correspondences.

[0054] Furthermore, a smaller displacement step size can reduce local image changes caused by lens distortion, optical aberrations, and changes in viewing angle, ensuring that the appearance features of the same marker point remain highly consistent in adjacent viewing angles. This is beneficial for improving the stability and accuracy of marker point matching and provides a reliable data foundation for the accurate solution of subsequent camera calibration parameters.

[0055] It should be noted that "half of the current viewpoint image field of view" in this embodiment is only a preferred implementation. In actual application, the displacement distance can also be adjusted according to factors such as the feature density of the scanned object, image resolution, and matching algorithm performance, as long as there is enough overlap between adjacent viewpoints and the requirement for repeated observation of the same-named marker points is met.

[0056] As a further optional embodiment, the step of identifying the same-named marker points corresponding to the same physical structure within the overlapping area of ​​adjacent frame images specifically includes: In the first frame image, multiple target image regions with preset pattern features are identified, and the center position of each target image region is determined as a candidate marker point; Based on the displacement information corresponding to the mobile platform, the predicted position of each candidate marker point in the second frame image is determined; Image matching is performed near the predicted location to determine the matching image region corresponding to each target image region; The center position of the matched image region is determined as the corresponding marker point in the second frame image; Based on the matching relationship between candidate markers in the first frame image and corresponding markers in the second frame image, the same-name markers corresponding to the same physical structure are determined.

[0057] In this embodiment, to establish the correspondence between the same physical structure from different viewpoints, a target image region with a preset pattern feature is first searched in the first frame image. The preset pattern feature can be a local texture feature, structural feature, grayscale distribution feature, or other image feature that can maintain stable recognition across multiple viewpoints. This embodiment does not require the target image region to correspond to a regular geometric structure, nor does it require the marker point to have pre-known spatial coordinates; it only requires that the corresponding region can be repeatedly recognized from different viewpoints. Subsequently, the center position of each target image region is determined as a candidate marker point, serving as a reference position for subsequent cross-viewpoint matching.

[0058] After obtaining candidate marker points, the positional changes of each candidate marker point in the second frame image are estimated based on the displacement information of the moving platform, and the corresponding predicted positions are determined. Since the displacement information of the moving platform reflects the relative positional relationship between the two viewpoints, the search range of candidate marker points can be limited to the vicinity of the predicted positions using this displacement information, thereby narrowing the matching search area, improving matching efficiency, and reducing the probability of false matches.

[0059] Subsequently, image matching processing is performed near the predicted location to find the matching image region with the highest similarity to the target image region in the first frame image. During the matching process, grayscale information, texture information, structural information, or feature description information can be comprehensively used to evaluate the similarity between the candidate region and the target image region, and the image region that meets the matching conditions is determined as the corresponding matching image region. Then, the center position of the matching image region is determined as the corresponding marker point in the second frame image.

[0060] Finally, based on the matching relationship established between candidate marker points in the first frame image and corresponding marker points in the second frame image, the same-name marker points corresponding to the same physical structure are determined. In this way, a stable correspondence between marker points can be established across multiple viewpoints, enabling unified association of observations of the same physical location in different images. This provides a reliable data foundation for subsequent construction of camera calibration models and solving for camera calibration parameters.

[0061] It should be noted that the image matching process can employ matching methods based on feature descriptors, template matching, image patch similarity, or other matching methods capable of establishing cross-viewpoint correspondences. This embodiment does not impose specific limitations on these methods. As long as marker points corresponding to the same physical structure under different viewpoints can be identified, they fall within the protection scope of this invention.

[0062] As a further optional embodiment, the step of jointly solving for the camera extrinsic parameters, camera intrinsic parameters, and marker point spatial structure parameters based on the camera calibration model to obtain the initial camera calibration parameters specifically includes: Obtain the initial intrinsic parameters of the camera, and normalize the image coordinates of the corresponding marker points based on the initial intrinsic parameters; The first solution model is constructed based on the normalized image coordinates, the displacement information of the moving platform, and the camera height information. The camera extrinsic parameters and the spatial structure parameters of the marker points are obtained through linear solution. A second solution model is constructed based on the camera extrinsic parameters and the spatial structure parameters of the marker points, and the camera intrinsic parameters are obtained through linear solution. The obtained camera extrinsic parameters, camera intrinsic parameters, and marker point spatial structure parameters are used as the initial camera calibration parameters.

[0063] In this embodiment, to obtain initial camera calibration parameters that describe the geometric and imaging relationships of the imaging system, the initial intrinsic parameters of the camera are first acquired. These initial intrinsic parameters can be derived from lens design parameters, equipment factory parameters, empirical parameters, or parameters obtained through pre-measurement, and are used to establish an initial correspondence between image coordinates and the camera imaging model. Subsequently, based on these initial intrinsic parameters, the image coordinates of corresponding marker points are normalized to reduce the impact of pixel scale differences on subsequent solution processes and to convert image observation information into a unified coordinate representation.

[0064] After normalization, a first solution model is constructed based on the normalized image coordinates, the displacement information of the moving platform, and the camera height information. The normalized image coordinates represent the observation results of corresponding marker points from various viewpoints; the displacement information of the moving platform represents the relative motion relationship between different viewpoints; and the camera height information represents the spatial relationship between the imaging system and the scanned object. By incorporating this information into a unified solution framework, the correlation between camera extrinsic parameters and marker point spatial structure parameters can be established, and initial estimates of both parameters can be obtained through linear solving.

[0065] After obtaining the camera extrinsic parameters and the spatial structure parameters of the marker points, a second solution model is constructed using these parameters. This second solution model describes the correspondence between the camera imaging parameters and the image observation results. By combining observation data from multiple perspectives, the camera intrinsic parameters are solved, thereby obtaining the estimated results of the camera intrinsic parameters.

[0066] Subsequently, the obtained camera extrinsic parameters, camera intrinsic parameters, and marker point spatial structure parameters are integrated to determine the initial camera calibration parameters. These initial camera calibration parameters include not only extrinsic parameters reflecting camera attitude information, but also intrinsic parameters reflecting camera imaging characteristics and marker point spatial structure parameters describing the spatial structure of the scanned object. Therefore, they can provide a relatively complete description of the geometric and imaging models of the current imaging system.

[0067] By first solving the camera's extrinsic parameters and the spatial structure parameters of the marker points, and then solving the camera's intrinsic parameters, the complex joint solution problem can be decomposed into multiple interrelated solution processes. This reduces the difficulty of the solution, improves the stability and convergence efficiency of parameter estimation, and provides a reliable initial parameter basis for subsequent optimization updates based on reprojection errors.

[0068] As a further optional embodiment, the step of constructing a reprojection model based on the camera calibration parameters and optimizing and updating the camera calibration parameters according to the reprojection error to obtain the target camera calibration parameters specifically includes: A reprojection model between the spatial coordinates of the marker points and the image coordinates is established based on the initial camera calibration parameters. The predicted image coordinates of each corresponding marker point are calculated using the reprojection model. The reprojection error is determined based on the difference between the predicted image coordinates and the corresponding actual image coordinates; The initial camera calibration parameters are iteratively optimized with the goal of minimizing the reprojection error. When the reprojection error meets the preset convergence condition, the optimized target camera calibration parameters are output.

[0069] In this embodiment, to further improve the accuracy of the camera calibration parameters, after obtaining the initial camera calibration parameters, a reprojection model between the spatial coordinates of the marker points and the image coordinates is established using the initial camera calibration parameters. The reprojection model describes the projection relationship of the marker points in space onto the image plane after the camera imaging process. This model allows for the prediction of the imaging position of the marker points in images from various viewpoints based on their spatial location.

[0070] Subsequently, the reprojection model is used to calculate the projection of each corresponding marker point to obtain the corresponding predicted image coordinates. The predicted image coordinates represent the theoretically correct position of the marker point in the image under the current camera calibration parameters. Simultaneously, the actual image coordinates of the corresponding marker points are obtained from the actual acquired image, and the predicted image coordinates are compared with the actual image coordinates to determine the difference between them. This difference reflects the degree of deviation between the current calibration parameters and the actual imaging situation and is defined as the reprojection error.

[0071] Furthermore, with minimizing the reprojection error as the optimization objective, the initial camera calibration parameters are iteratively optimized. During the optimization process, the camera calibration parameters are updated based on the currently calculated reprojection error, and the reprojection model is reconstructed using the updated parameters. The predicted image coordinates and the corresponding reprojection error are then recalculated. By continuously repeating the above process, the reprojection error is gradually reduced, thereby enabling the camera calibration parameters to continuously approximate the true parameters of the actual imaging system.

[0072] When the reprojection error drops below a preset threshold, or the error change between two consecutive iterations is less than a preset convergence threshold, the current optimization process can be considered to have reached the preset convergence condition. At this point, the iteration stops, and the optimized camera extrinsic parameters, camera intrinsic parameters, and marker point spatial structure parameters are output as the target camera calibration parameters.

[0073] By introducing an optimization update mechanism based on reprojection error, we can not only make full use of the geometric constraints in multi-view observation data, but also effectively correct the parameter deviations generated in the initial solution process. This makes the final target camera calibration parameters more consistent with the imaging laws of the actual imaging system, thereby improving the accuracy and stability of the camera calibration results and providing a reliable parameter basis for subsequent high-precision measurement, defect detection and image analysis applications.

[0074] As a further optional embodiment, the step of iteratively optimizing the initial camera calibration parameters with the goal of minimizing the reprojection error specifically includes: The camera intrinsic parameters, the camera extrinsic parameters, the lens distortion parameters, and the marker point spatial structure parameters are used as variables to be optimized. Construct a nonlinear optimization model with reprojection error as the objective function; The nonlinear least squares optimization algorithm is used to jointly solve the variables to be optimized in order to obtain the target camera calibration parameters.

[0075] In this embodiment, to further improve the accuracy of solving the camera calibration parameters, the camera intrinsic parameters, camera extrinsic parameters, lens distortion parameters, and marker point spatial structure parameters are all treated as variables to be optimized and solved jointly under a unified optimization framework. The camera intrinsic parameters describe the camera's own imaging characteristics, the camera extrinsic parameters describe the spatial relationship between the camera and the scanned object, the lens distortion parameters characterize the imaging distortion introduced by the optical system, and the marker point spatial structure parameters describe the spatial distribution of each marker point. Since these parameters are interrelated and jointly affect the final projection result, joint optimization can more accurately reflect the geometric relationships and imaging laws of the actual imaging system.

[0076] Furthermore, based on the actual image coordinates of the corresponding markers at each viewpoint and the predicted image coordinates calculated based on the current parameters, a nonlinear optimization model with reprojection error as the objective function is constructed. The objective function measures the overall deviation between the predicted and actual observation results for all corresponding markers at each observation viewpoint. By applying uniform constraints to all observation data, the optimization process can fully utilize the geometric consistency information in the multi-view data.

[0077] Subsequently, a nonlinear least squares optimization algorithm is used to jointly solve for the variables to be optimized. During the optimization process, the objective function value and parameter update direction are calculated based on the current iteration results, and adjustments are made to each variable to be optimized to continuously reduce the objective function. As the iteration process continues, the deviation between the predicted image coordinates and the actual image coordinates gradually decreases, and the reprojection error gradually converges, thereby obtaining more accurate parameter estimation results.

[0078] Because camera intrinsic parameters, camera extrinsic parameters, lens distortion parameters, and marker point spatial structure parameters are incorporated into the same optimization model for coordinated adjustment, the coupling relationships between various parameters can be fully considered, avoiding the error accumulation problem caused by optimizing only one type of parameter. Furthermore, by uniformly evaluating all observation data through reprojection errors, multi-view observation information can be effectively utilized to globally constrain the parameters, thereby improving the stability and reliability of the parameter solution.

[0079] Finally, once the optimization process meets the preset convergence conditions, the optimized target camera calibration parameters are output. Compared with the initial camera calibration parameters, the obtained target camera calibration parameters can more accurately describe the imaging characteristics, spatial geometric relationships, and lens distortion characteristics of the actual imaging system, thereby further improving the camera calibration accuracy and providing more reliable basic parameters for subsequent applications such as high-precision detection, dimensional measurement, and image analysis.

[0080] In summary, a minimal embodiment of the present invention is as follows: First, the moving platform is controlled to move multiple times within a plane, and an image frame is acquired after each movement, while simultaneously recording the displacement information relative to the initial position. A certain overlap is maintained between adjacent viewpoints, allowing some marker points on the scanned object to reappear under multiple viewpoints, thus forming the data foundation required for subsequent matching of corresponding points. The scanned object can be a process object with stable patterns or microstructure features, such as a wafer surface area, as long as its local pattern can be stably identified under different viewpoints. Specifically, the platform is first controlled to move within a two-dimensional plane, and the coordinates of the platform center relative to the initial position in the platform coordinate system are recorded after each movement.

[0081] in,i Image sequence number j This is the calibration point number.

[0082] Subsequently, corresponding marker points with the same physical structure are identified within the overlapping region of adjacent frames. Specifically, several target image regions with stable pattern features are first found in the first frame image, and their center positions are used as candidate marker points. Then, the approximate positions of these candidate marker points in the next frame are estimated based on the displacement information of the moving platform, and image matching is performed near these positions to obtain the corresponding marker points in the second frame. This establishes the correspondence between the corresponding marker points in the first and second frames and can be further extended to all viewpoints. Specifically, after each movement, a frame image is acquired using the area array mode of a line scan camera, and corresponding marker points are searched within the overlapping region of adjacent images. For the j-th marker point, the image coordinates in the i-th viewpoint are:

[0083] in, i Image sequence number j This is the calibration point number.

[0084] After establishing the correspondence between corresponding marker points, a camera calibration model is constructed based on displacement information and the image coordinates of these marker points. This model unifies platform displacement, marker point spatial positions, and image observation results into a single geometric constraint framework. Then, the camera extrinsic parameters, camera intrinsic parameters, and marker point spatial structure parameters are jointly solved according to the calibration model to obtain initial camera calibration parameters. First, the image coordinates of the corresponding marker points are normalized using the initial intrinsic parameters. Then, a linear solution is performed based on the normalized coordinates, platform displacement information, and camera height information to obtain the extrinsic and spatial structure parameters, followed by the calculation of the intrinsic parameters. Specifically, to eliminate the influence of pixel scale, the image coordinates are first normalized using the currently estimated intrinsic parameters to obtain undistorted normalized image coordinates. :

[0085] in, K This is the current estimated intrinsic parameter matrix. This represents the inverse distortion process. Initially, the magnification is based on the optical system design value, the principal point coordinates are based on the initial estimated value, and the distortion parameters are based on the preset value.

[0086] Let the camera height be h The rotation matrix is R The projection direction of the spatial structure is M j .

[0087]

[0088] Solving the rotation matrix using least squares R and the projection direction of the spatial structure M j .

[0089] After obtaining the initial camera calibration parameters, a reprojection model is further constructed and optimized. The basic idea is to project the spatial coordinates of the marker points onto the image plane using the current camera parameters to obtain the predicted image coordinates; then compare these predicted coordinates with the actually observed image coordinates to calculate the reprojection error; finally, iteratively optimize the parameters with the goal of minimizing the reprojection error until the convergence condition is met, and output the target camera calibration parameters. A global nonlinear optimization process with the goal of minimizing the reprojection error is adopted, and it can be implemented using LM-type algorithms.

[0090] Specifically, firstly, based on the obtained rotation matrix R and the projection direction of the spatial structure M j Reproject to obtain the predicted coordinates calculated based on the current parameters. :

[0091] Then, distortion mapping is performed to obtain the mapped coordinates. :

[0092] Finally mapped to pixel coordinates:

[0093] The objective function is obtained as follows:

[0094] in, Indicates the actual observed coordinates. This represents the predicted coordinates calculated based on the current parameters. During the optimization process, intrinsic parameters, extrinsic parameters, distortion parameters, and spatial structure parameters of the marker points are updated simultaneously, and the LM algorithm is used for solving.

[0095] The linear array camera calibration device for scanning feature points provided by the present invention will be described below, such as... Figure 3 As shown, the linear scan camera calibration device for scanning feature points described below and the linear scan camera calibration method for scanning feature points described above can be referred to in correspondence.

[0096] A calibration device for a linear array camera that scans feature points, comprising: Information acquisition module 310 is used to acquire displacement information corresponding to the multiple movements of the mobile platform in the plane, and to acquire an image frame after each movement, the image containing multiple marker points on the scanned object; The marker recognition module 320 is used to identify the same-name marker points corresponding to the same physical structure in the overlapping area of ​​adjacent frame images. The same-name marker points are marker points that correspond to the same physical location from different viewpoints. Model building module 330 is used to build a camera calibration model based on the displacement information and the image coordinates of the corresponding marker points; The parameter solving module 340 is used to jointly solve the camera extrinsic parameters, camera intrinsic parameters, and marker point spatial structure parameters according to the camera calibration model to obtain the initial camera calibration parameters. The optimization and update module 350 is used to construct a reprojection model based on the camera calibration parameters, and optimize and update the camera calibration parameters according to the reprojection error to obtain the target camera calibration parameters.

[0097] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a linear scan camera calibration method for scanning feature points, the method including: The displacement information corresponding to the multiple movements of the mobile platform in the plane is obtained, and an image frame is captured after each movement. The image contains multiple marker points on the scanned object. Within the overlapping area of ​​adjacent frame images, identify the same-named marker points corresponding to the same physical structure. The same-named marker points are marker points that correspond to the same physical location from different viewpoints. A camera calibration model is established based on the displacement information and the image coordinates of the corresponding marker points; Based on the camera calibration model, the camera extrinsic parameters, camera intrinsic parameters, and marker point spatial structure parameters are jointly solved to obtain the initial camera calibration parameters. A reprojection model is constructed based on the camera calibration parameters, and the camera calibration parameters are optimized and updated according to the reprojection error to obtain the target camera calibration parameters.

[0098] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0099] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the linear scan camera calibration method for scanning feature points provided by the above methods, the method comprising: The displacement information corresponding to the multiple movements of the mobile platform in the plane is obtained, and an image frame is captured after each movement. The image contains multiple marker points on the scanned object. Within the overlapping area of ​​adjacent frame images, identify the same-named marker points corresponding to the same physical structure. The same-named marker points are marker points that correspond to the same physical location from different viewpoints. A camera calibration model is established based on the displacement information and the image coordinates of the corresponding marker points; Based on the camera calibration model, the camera extrinsic parameters, camera intrinsic parameters, and marker point spatial structure parameters are jointly solved to obtain the initial camera calibration parameters. A reprojection model is constructed based on the camera calibration parameters, and the camera calibration parameters are optimized and updated according to the reprojection error to obtain the target camera calibration parameters.

[0100] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a linear scan camera calibration method for scanning feature points provided by the methods described above, the method comprising: The displacement information corresponding to the multiple movements of the mobile platform in the plane is obtained, and an image frame is captured after each movement. The image contains multiple marker points on the scanned object. Within the overlapping area of ​​adjacent frame images, identify the same-named marker points corresponding to the same physical structure. The same-named marker points are marker points that correspond to the same physical location from different viewpoints. A camera calibration model is established based on the displacement information and the image coordinates of the corresponding marker points; Based on the camera calibration model, the camera extrinsic parameters, camera intrinsic parameters, and marker point spatial structure parameters are jointly solved to obtain the initial camera calibration parameters. A reprojection model is constructed based on the camera calibration parameters, and the camera calibration parameters are optimized and updated according to the reprojection error to obtain the target camera calibration parameters.

[0101] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0102] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A calibration method for a linear scan camera that scans feature points, applied to an imaging system including a linear scan camera and a moving platform, characterized in that, include: The displacement information corresponding to the multiple movements of the mobile platform in the plane is obtained, and an image frame is captured after each movement. The image contains multiple marker points on the scanned object. Within the overlapping area of ​​adjacent frame images, identify the same-named marker points corresponding to the same physical structure. The same-named marker points are marker points that correspond to the same physical location from different viewpoints. A camera calibration model is established based on the displacement information and the image coordinates of the corresponding marker points; Based on the camera calibration model, the camera extrinsic parameters, camera intrinsic parameters, and marker point spatial structure parameters are jointly solved to obtain the initial camera calibration parameters. A reprojection model is constructed based on the camera calibration parameters, and the camera calibration parameters are optimized and updated according to the reprojection error to obtain the target camera calibration parameters. The step of jointly solving for the camera extrinsic parameters, camera intrinsic parameters, and marker point spatial structure parameters based on the camera calibration model to obtain the initial camera calibration parameters specifically includes: Obtain the initial intrinsic parameters of the camera, and normalize the image coordinates of the corresponding marker points based on the initial intrinsic parameters; The first solution model is constructed based on the normalized image coordinates, the displacement information of the moving platform, and the camera height information. The camera extrinsic parameters and the spatial structure parameters of the marker points are obtained through linear solution. A second solution model is constructed based on the camera extrinsic parameters and the spatial structure parameters of the marker points, and the camera intrinsic parameters are obtained through linear solution. The obtained camera extrinsic parameters, camera intrinsic parameters, and marker point spatial structure parameters are used as the initial camera calibration parameters.

2. The linear array camera calibration method for scanning feature points according to claim 1, characterized in that, The step of acquiring displacement information corresponding to multiple movements of the mobile platform within the plane, and capturing an image frame after each movement, specifically includes: The mobile platform is controlled to make multiple position adjustments in a two-dimensional plane according to a preset motion trajectory; After each position adjustment, the displacement coordinates of the mobile platform relative to the initial position are obtained as the displacement information corresponding to the current viewpoint; and the line scan camera is controlled to acquire images of the scanned object to obtain the image corresponding to the current viewpoint. The position adjustment, displacement acquisition, and image acquisition process is repeated to obtain displacement information and images corresponding to multiple different viewpoints; The images corresponding to two adjacent position adjustments have overlapping areas, so that at least some of the markers on the scanned object can be repeatedly observed from multiple perspectives.

3. The linear array camera calibration method for scanning feature points according to claim 2, characterized in that, The displacement distance corresponding to two consecutive position adjustments is less than half of the current viewpoint image's field of view.

4. The linear array camera calibration method for scanning feature points according to claim 1, characterized in that, The step of identifying the same-named marker points corresponding to the same physical structure within the overlapping area of ​​adjacent frame images specifically includes: In the first frame image, multiple target image regions with preset pattern features are identified, and the center position of each target image region is determined as a candidate marker point; Based on the displacement information corresponding to the mobile platform, the predicted position of each candidate marker point in the second frame image is determined; Image matching is performed near the predicted location to determine the matching image region corresponding to each target image region; The center position of the matched image region is determined as the corresponding marker point in the second frame image; Based on the matching relationship between candidate markers in the first frame image and corresponding markers in the second frame image, the same-name markers corresponding to the same physical structure are determined.

5. The linear array camera calibration method for scanning feature points according to claim 1, characterized in that, The step of constructing a reprojection model based on the camera calibration parameters and optimizing and updating the camera calibration parameters according to the reprojection error to obtain the target camera calibration parameters specifically includes: A reprojection model between the spatial coordinates of the marker points and the image coordinates is established based on the initial camera calibration parameters. The predicted image coordinates of each corresponding marker point are calculated using the reprojection model. The reprojection error is determined based on the difference between the predicted image coordinates and the corresponding actual image coordinates; The initial camera calibration parameters are iteratively optimized with the goal of minimizing the reprojection error. When the reprojection error meets the preset convergence condition, the optimized target camera calibration parameters are output.

6. The linear array camera calibration method for scanning feature points according to claim 5, characterized in that, The step of iteratively optimizing the initial camera calibration parameters with the goal of minimizing the reprojection error specifically includes: The camera intrinsic parameters, the camera extrinsic parameters, the lens distortion parameters, and the marker point spatial structure parameters are used as variables to be optimized. Construct a nonlinear optimization model with reprojection error as the objective function; The nonlinear least squares optimization algorithm is used to jointly solve the variables to be optimized in order to obtain the target camera calibration parameters.

7. A calibration device for a linear scan camera that scans feature points, applied to an imaging system including a linear scan camera and a moving platform, characterized in that, include: The information acquisition module is used to acquire displacement information corresponding to the multiple movements of the mobile platform in the plane, and to acquire an image frame after each movement, the image containing multiple marker points on the scanned object; The marker recognition module is used to identify the same-name marker points corresponding to the same physical structure in the overlapping area of ​​adjacent frame images. The same-name marker points are marker points that correspond to the same physical location from different viewpoints. The model building module is used to build a camera calibration model based on the displacement information and the image coordinates of the corresponding marker points; The parameter solving module is used to jointly solve the camera extrinsic parameters, camera intrinsic parameters, and marker point spatial structure parameters according to the camera calibration model to obtain the initial camera calibration parameters. The optimization and update module is used to construct a reprojection model based on the camera calibration parameters, and optimize and update the camera calibration parameters according to the reprojection error to obtain the target camera calibration parameters. The step of jointly solving for the camera extrinsic parameters, camera intrinsic parameters, and marker point spatial structure parameters based on the camera calibration model to obtain the initial camera calibration parameters specifically includes: Obtain the initial intrinsic parameters of the camera, and normalize the image coordinates of the corresponding marker points based on the initial intrinsic parameters; The first solution model is constructed based on the normalized image coordinates, the displacement information of the moving platform, and the camera height information. The camera extrinsic parameters and the spatial structure parameters of the marker points are obtained through linear solution. A second solution model is constructed based on the camera extrinsic parameters and the spatial structure parameters of the marker points, and the camera intrinsic parameters are obtained through linear solution. The obtained camera extrinsic parameters, camera intrinsic parameters, and marker point spatial structure parameters are used as the initial camera calibration parameters.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the linear array camera calibration method for scanning feature points as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the linear scan camera calibration method for scanning feature points as described in any one of claims 1 to 6.

Images