Multi-view-angle-oriented external parameter calibration and plane estimation method

Abstract

The invention belongs to the field of computer vision, particularly relates to a multi-view-angle-oriented external parameter calibration and plane estimation method, and aims to solve the problems of poor reliability and complex process during camera external parameter calibration and plane estimation in the prior art. The method comprises the following steps: acquiring a target image; the target images are preprocessed, and checkerboard angular points are extracted from the preprocessed target images; according to the checkerboard angular points in each target image, simultaneously establishing a target equation set of each target image, and solving the target equation set by adopting a preset optimization algorithm to obtain external parameters of the target camera; and determining a plane equation of each to-be-determined plane according to the checkerboard angular points in each target image. Based on the method, the checkerboard images shot from different visual angles are integrated, the plane equation is automatically calculated while the extrinsic parameters of the camera are accurately solved, and the plane estimation process is remarkably simplified.

Description

Technical Field

[0001] The invention belongs to the field of computer vision, and in particular relates to an extrinsic parameter calibration and plane estimation method for multiple viewing angles. Background Art

[0002] In the field of image processing technology, camera calibration is a key link, which is widely used in target tracking, image fusion, 3D reconstruction and robot navigation. The purpose of camera calibration is to obtain the camera's intrinsic parameters (such as focal length, principal point, etc.) and extrinsic parameters (such as rotation and translation matrix) in order to convert points in the image coordinate system into points in the world coordinate system, which will be used for subsequent 3D reconstruction, SLAM and other tasks.

[0003] Existing camera calibration solutions do not work well when processing checkerboard images with large angles of view, small proportions, and low resolutions. Taking the OpenCV library as an example, due to the sensitivity of its corner detection algorithm to image quality, some corner points are often inaccurately positioned. There are deficiencies in the accuracy of corner detection, which directly affects the accuracy and reliability of subsequent calibration.

[0004] At the same time, in this field, existing multi-view ground plane estimation methods (such as MAGE) mostly complete plane estimation through manual point selection and point cloud reconstruction, which makes point cloud reconstruction rely on a large number of high-quality feature points, and puts high requirements on images. At the same time, the process of reconstructing point clouds relying on a large number of feature points is complicated and time-consuming, and needs to be combined with manual annotation. The quality of feature points directly affects the reconstruction accuracy, which in turn affects the effect of plane estimation. Summary of the invention

[0005] In order to solve the above-mentioned problems in the prior art, namely, the technical problems of poor reliability and complicated process in the prior art when performing camera extrinsic calibration and plane estimation, the first aspect of the present invention proposes a multi-view extrinsic calibration and plane estimation method, comprising: Acquire a target image, wherein the target image is obtained by sequentially capturing images of a plurality of undetermined planes on which checkerboards are placed, using a target camera based on a plurality of preset camera positions; Preprocessing the target image and extracting checkerboard corner points from each preprocessed target image, wherein the preprocessing includes graying, filtering and edge detection preprocessing; According to the checkerboard corner points in each target image, a target equation group of each target image is jointly established, and a preset optimization algorithm is used to solve the target equation group to obtain the external parameters of the target camera, wherein the target equation group is constructed based on the mapping relationship between the world coordinate system and the camera coordinate system where the checkerboard corner points in each target image are located, and the preset optimization algorithm solves the minimum value of a preset objective function by iteratively updating algorithm parameters to optimize the external parameters of the target camera, and the objective function is used to indicate the residual between the predicted pixel coordinates and the actual pixel coordinates; According to the checkerboard corner points in each target image, the plane equations of each undetermined plane are determined.

[0006] In some preferred embodiments, the method of combining a set of equations of multiple target images according to the checkerboard corner points in each target image, and solving the target set of equations using a preset optimization algorithm to obtain the external parameters of the target camera includes: Determine the pixel coordinates and world coordinates of the checkerboard corner points in each target image; According to the pixel coordinates and world coordinates of the corner points of the chessboard, a target equation group after the multi-graph combination is determined, and the target equation group is solved jointly to obtain initial external parameters; The initial extrinsic parameters are optimized using the Levenberg-Marquardt least squares optimization algorithm to obtain the extrinsic parameters of the target camera.

[0007] In some preferred embodiments, the pixel coordinates and world coordinates of the checkerboard corner points in each target image are expressed in homogeneous coordinates as follows: ; The target equation group after the multi-graph combination is: ; In the formula, is the parameter to be solved, is the external parameter of the target camera.

[0008] In some preferred implementations, when the pixel coordinates of the checkerboard corner points in multiple target images and the corresponding world coordinates are combined, the contribution value of each target image to the combined result is balanced by setting the weight corresponding to each target image.

[0009] In some preferred embodiments, the method further comprises: Determine the reprojection error of each undetermined plane before and after the external parameter calibration respectively; According to the reprojection error, it is determined whether the extrinsic parameters of the target camera meet a preset accuracy threshold.

[0010] In some preferred embodiments, the checkerboard corner points in any undetermined plane The world coordinates satisfy: ; Passing point The normal vector is , then point The plane equation of the undetermined plane is: ; in, , the normal vector It is calculated from the three non-collinear corner points in the chessboard where the plane is placed; The homogeneous coordinate form of the normal vector n is , then point The normal vector n satisfies: ; Assume that the checkerboard corner points in another undetermined plane are ,but The world coordinates satisfy: ; in, Represents the position transformation matrix of a set of checkerboard corner points in the world coordinate system; Passing point The normal vector satisfy: ; Assumed normal vector The homogeneous coordinate form of is: , then point The plane equation of the undetermined plane is: .

[0011] In some preferred embodiments, the step of determining the plane equations of each undetermined plane according to the checkerboard corner points in each target image includes: Determine the coordinates of the checkerboard corner points in the world coordinate system as follows: ; In the formula, , is the coordinate of each point in the world coordinate system when the chessboard is placed on the kth plane; In the chessboard corner point coordinate collection A fitting plane, wherein the fitting plane corresponds to a set of plane normal vectors and offsets to be solved; exist Randomly select three non-collinear points in , to construct two direction vectors to solve the plane normal vector and offset corresponding to the fitting plane, and the solution process satisfies: ; In the formula, ; ; Points The x-coordinate, y-coordinate and z-coordinate of exist In the remaining point set, traverse each point in turn Distance from the fitting plane , and set the distance threshold to determine the Is it less than the distance threshold? is less than the distance threshold, the current point is determined is a point in the plane; After the traversal is completed, The set with the largest number of interior points is determined to optimize the parameters of the fitting plane until the final plane equation is obtained.

[0012] In a second aspect, the present invention further proposes a multi-view extrinsic calibration and plane estimation system, comprising: A data acquisition module is used to acquire a target image, wherein the target image is acquired by sequentially acquiring images of a plurality of undetermined planes on which checkerboards are placed, using a target camera based on a plurality of preset camera positions; A preprocessing module, used for preprocessing the target image and extracting checkerboard corner points in each preprocessed target image, wherein the preprocessing includes graying, filtering and edge detection preprocessing; An extrinsic parameter solving module is used to jointly establish a target equation group of each target image according to the checkerboard corner points in each target image, and solve the target equation group using a preset optimization algorithm to obtain the extrinsic parameters of the target camera, wherein the target equation group is constructed based on the mapping relationship between the world coordinate system and the camera coordinate system where the checkerboard corner points in each target image are located, and the preset optimization algorithm solves the minimum value of a preset objective function by iteratively updating algorithm parameters to optimize the extrinsic parameters of the target camera, and the objective function is used to indicate the residual between the predicted pixel coordinates and the actual pixel coordinates; The plane estimation module is used to determine the plane equations of each undetermined plane according to the checkerboard corner points in each target image.

[0013] In a third aspect, the present invention further provides an electronic device, comprising: at least one processor; and a memory communicatively connected to at least one of the processors; wherein, The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the multi-view oriented external parameter calibration and plane estimation method as described in the first aspect.

[0014] In a fourth aspect, the present invention further proposes a computer-readable storage medium, characterized in that the computer-readable storage medium stores computer instructions, and the computer instructions are used to be executed by the computer to implement the multi-view extrinsic parameter calibration and plane estimation method as described in the first aspect.

[0015] Beneficial effects of the present invention: (1) Compared with the traditional single-image calibration method, the method of the present application enhances the robustness of the calibration process by combining the corner points of multi-view chessboards. In practical applications, images may be affected by a variety of factors such as large angles of view, small proportions, and low resolution. The method of the present application uses multi-image joint calibration, which is equivalent to combining multiple chessboards into a large chessboard, increasing the number of corner points, expanding the coverage area, and enriching the perspective mode. It effectively handles the impact of these factors and makes the calibration results more stable and reliable. Different weights can even be assigned according to the different chessboards in the camera imaging area (center or edge), which can alleviate the impact of imaging quality to a certain extent.

[0016] (2) Compared with the traditional calibration method, the method of the present application separates the calibration process of internal and external parameters. Generally speaking, the internal parameters of a camera are fixed after leaving the factory, so they can be accurately calibrated in a laboratory environment. The method of the present application calibrates the external parameters under the condition that the internal parameters are known, which can eliminate the error interference of the internal parameter calibration. At the same time, the traditional calibration method actually only uses one image in the process of solving the camera external parameters. The present method can further improve the accuracy of the external parameter calibration by using multiple checkerboard images from different perspectives.

[0017] (3) Reduce manual annotation and reliance on feature points. Traditional plane estimation usually relies on feature point matching to construct point clouds. The quality and quantity of feature points will significantly affect the performance of the algorithm. The checkerboard method directly provides high-quality feature points through geometric structures without the need for point cloud reconstruction or time-consuming matching processes. This method combines the calibration and plane estimation steps into one, fully automating the process and avoiding the manual annotation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings: Figure 1 It is a flowchart of a multi-view extrinsic parameter calibration and plane estimation method proposed in an embodiment of the present invention; Figure 2It is a structural diagram of a computer system of a server proposed in an embodiment of the present invention and used to implement the method and system embodiments of the present application. DETAILED DESCRIPTION

[0019] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0020] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0021] Reference Figure 1 The first embodiment of the present invention provides a multi-view extrinsic parameter calibration and plane estimation method, comprising: Step S10, obtaining a target image, wherein the target image is obtained by sequentially capturing images of a plurality of undetermined planes on which checkerboards are placed, using a target camera based on a plurality of preset camera positions; In this embodiment, the target image refers to a plurality of chessboard images obtained by sequentially placing chessboards for extrinsic parameter calibration on different planes and photographing them from a plurality of fixed preset positions using a target camera with to-be-determined extrinsic parameters.

[0022] It should be noted that, in this embodiment, based on the division of extrinsic parameter calibration and plane estimation, the above-mentioned target cameras at the execution entity level include at least: a single target camera for extrinsic parameter calibration, and multiple target cameras for plane estimation. This embodiment does not make further naming distinctions on the number of target cameras, that is, the target cameras in this embodiment are used to characterize all cameras to be used for subsequent camera calibration and plane estimation.

[0023] Based on this understanding, the multiple preset positions in this embodiment refer to pre-set, fixed camera shooting positions. Specifically, for camera extrinsic calibration, the multiple preset positions can be regarded as corresponding one-to-one to each target camera to be calibrated, and each target camera to be calibrated is fixed at its corresponding position, and multiple shots are taken when the chessboard moves; for plane estimation, the multiple preset positions can be regarded as a sequence set corresponding to the fixed position of each target camera.

[0024] In this embodiment, it is assumed that the size of the chessboard and the positions of each corner point in the real world are known. Therefore, this embodiment obtains multiple samples of camera extrinsic calibration by taking multiple images of the same chessboard at different viewing angles for use in subsequent steps, which can enhance the robustness of the calibration process.

[0025] Step S20, preprocessing the target image and extracting checkerboard corner points from each preprocessed target image, wherein the preprocessing includes graying, filtering and edge detection preprocessing; In this embodiment, the purpose of preprocessing the acquired target image is to improve the image quality so as to facilitate the subsequent corner point extraction. The preprocessing in this embodiment includes three parts: grayscale processing, filtering processing and edge detection preprocessing; Grayscale processing is used to convert color images into grayscale images to simplify image data and reduce the amount of calculation when extracting checkerboard corner points, while retaining sufficient structural information for corner point detection; Filtering is used to filter grayscale images to remove noise and spots, while making the processed image smoother, which is helpful for subsequent edge detection and corner location.

[0026] Edge detection is used to identify edge information in an image using a preset edge detection algorithm (this embodiment does not limit the specific algorithm). It is easy to understand that since the corners of the chessboard are usually located at the intersection of the edges, edge detection can effectively save the amount of calculation required for corner point extraction.

[0027] Step S30, according to the checkerboard corner points in each target image, jointly establish a target equation group of each target image, and adopt a preset optimization algorithm to solve the target equation group to obtain the external parameters of the target camera, wherein the target equation group is constructed according to the mapping relationship between the world coordinate system and the camera coordinate system where the checkerboard corner points in each target image are located, and the preset optimization algorithm solves the minimum value of a preset objective function by iteratively updating algorithm parameters to optimize the external parameters of the target camera, and the objective function is used to indicate the residual between the predicted pixel coordinates and the actual pixel coordinates; The preset optimization algorithm in this embodiment can be a Levenberg-Marquardt least squares optimization algorithm, which combines the advantages of the gradient descent method and the Gauss-Newton method to solve the nonlinear least squares problem in this field. The nonlinear least squares problem is usually expressed as finding a parameter vector x such that the sum of the squared errors between the model prediction value and the observed value is minimized.

[0028] Specifically, the above process includes: Determine the pixel coordinates and world coordinates of the checkerboard corner points in each target image; According to the pixel coordinates and world coordinates of the corner points of the chessboard, a target equation group after the multi-graph combination is determined, and the target equation group is solved jointly to obtain initial external parameters; The initial extrinsic parameters are optimized using the Levenberg-Marquardt least squares optimization algorithm to obtain the extrinsic parameters of the target camera.

[0029] Further, it is known that each image includes: ; In each image, the pixel coordinates of the checkerboard corners and the world coordinates (both expressed in homogeneous coordinates) are: ; Let the camera intrinsic parameter be M, and take the world coordinate system of the checkerboard corner point corresponding to image1 as the reference world coordinate system. The relationship between the world coordinates of the remaining points and the world coordinates of the checkerboard corner point corresponding to image1 can be expressed as: ; in, Represents the position transformation of the corner points of the chessboard in the world coordinate system, satisfying: ; in is the introduced rotation matrix variable, with three degrees of freedom, which can be simplified into three variables by Rodriguez, is the displacement variable, with three degrees of freedom.

[0030] Finally, the system of equations after multiple graphs are combined can be expressed as: ; In the formula, is the parameter to be solved, is the external parameter of the target camera.

[0031] In this embodiment, when the pixel coordinates of the checkerboard corner points in a plurality of target images and the corresponding world coordinates are combined, the contribution value of each target image to the combined result is balanced by setting the weight corresponding to each target image.

[0032] Compared with the traditional single-image calibration method, the method of the present application enhances the robustness of the calibration process by combining multi-view checkerboard corner points. In practical applications, images may be affected by a variety of factors such as large-angle viewing angle, small proportion, and low resolution. The method of the present application effectively handles the impact of these factors through multi-image joint calibration, making the calibration result more stable and reliable.

[0033] Step S40, determining the plane equations of each undetermined plane according to the checkerboard corner points in each target image.

[0034] It is easy to understand that in the prior art, the external parameter calibration and plane estimation solutions for multiple views generally include: 1. Calibrate external parameters; 2. Manually mark points on the ground / multi-planes of each view; 3. Establish equations for the ground / multi-planes. This approach is relatively cumbersome because points need to be manually marked after the external parameter calibration is completed; In this embodiment, multi-image joint calibration is performed when calibrating external parameters (as described above). During calibration, the chessboard is placed on different planes respectively. After the calibration is completed, the equations of each plane can be obtained at the same time, eliminating the need for manual point marking.

[0035] That is, the present embodiment only requires a calibration process to obtain the camera external parameters and the plane equation, thus simplifying the traditional process.

[0036] In this embodiment, the plane equation of the undetermined plane satisfies: ; In the formula, , the normal vector It is calculated from the three non-collinear corner points in the chessboard where the plane is placed.

[0037] This embodiment provides two implementation methods to determine the plane equations of each unknown plane. On the first aspect, the plane equations can be calculated using the world coordinate system and the transformation matrix T of different planes, which is the basic scheme of this embodiment. On the other hand, in order to eliminate the interference of noise and outliers, the plane equations can be calculated using the three-dimensional coordinates of the corner points of the chessboard and the RANSAC algorithm, which is another optional scheme of this embodiment.

[0038] In the first aspect, the plane equations are calculated using the world coordinate system and the transformation matrix T of different planes, including: Assume that the corner points of the chessboard are placed on the ground The world coordinates are: ; Passing point The normal vector is , then point The plane equation of the undetermined plane is: ; in, , the normal vector It is calculated from the three non-collinear corner points in the chessboard where the plane is placed; The homogeneous coordinate form of the normal vector n is ,point With the vector n satisfying: ; Assume that the checkerboard corner points in another undetermined plane are , The world coordinates satisfy: ; in, Represents the position transformation matrix of a set of checkerboard corner points in the world coordinate system; Passing point The normal vector satisfy: ; Assumed normal vector The homogeneous coordinate form of is: , then point The plane equation of the undetermined plane is: .

[0039] The transformations of other planes can be deduced in the same way.

[0040] In this embodiment, the plane equation is calculated using the three-dimensional coordinates of the corner points of the chessboard and the RANSAC algorithm, specifically including: Determine the coordinates of the checkerboard corner points in the world coordinate system as follows: ; In the formula, , is the coordinate of each point in the world coordinate system when the chessboard is placed on the kth plane; In the chessboard corner point coordinate collection A fitting plane, wherein the fitting plane corresponds to a set of plane normal vectors and offsets to be solved; exist Randomly select three non-collinear points in , to construct two direction vectors to solve the plane normal vector and offset corresponding to the fitting plane, and the solution process satisfies: ; In the formula, ; ; Points The x-coordinate, y-coordinate and z-coordinate of exist In the remaining point set, traverse each point in turn Distance from the fitting plane , and set the distance threshold to determine the Is it less than the distance threshold? is less than the distance threshold, the current point is determined is a point in the plane; After the traversal is completed, The set with the largest number of interior points is determined to optimize the parameters of the fitting plane until the final plane equation is obtained.

[0041] Based on the above embodiments, the present application proposes a multi-view extrinsic calibration and plane estimation method, which integrates chessboard images taken from different perspectives to construct a set of parallel equations to achieve accurate solution of camera extrinsics. Compared with the prior art, the present method enhances the robustness of the calibration process by using multi-view data, and improves the calibration accuracy by joint solution. At the same time, the present invention also proposes a multi-view plane estimation method based on chessboards, which can automatically calculate the equation of the plane by placing chessboards on the plane and combining multi-view corner point information, thereby significantly simplifying the process of plane estimation. This method reduces the dependence on feature point matching, avoids the tedious process of manually marking ground areas, improves the automation and efficiency of plane estimation, and enhances robustness in complex scenes. By calibrating the plane with chessboards, not only a stable geometric benchmark is provided for camera extrinsic calibration, but also a solid foundation is laid for subsequent detection and tracking tasks.

[0042] The second embodiment of the present application provides a multi-view extrinsic calibration and plane estimation system, including: A data acquisition module is used to acquire a target image, wherein the target image is acquired by sequentially acquiring images of a plurality of undetermined planes on which checkerboards are placed, using a target camera based on a plurality of preset camera positions; A preprocessing module, used for preprocessing the target image and extracting checkerboard corner points in each preprocessed target image, wherein the preprocessing includes graying, filtering and edge detection preprocessing; An extrinsic parameter solving module is used to jointly establish a target equation group of each target image according to the checkerboard corner points in each target image, and solve the target equation group using a preset optimization algorithm to obtain the extrinsic parameters of the target camera, wherein the target equation group is constructed based on the mapping relationship between the world coordinate system and the camera coordinate system where the checkerboard corner points in each target image are located, and the preset optimization algorithm solves the minimum value of a preset objective function by iteratively updating algorithm parameters to optimize the extrinsic parameters of the target camera, and the objective function is used to indicate the residual between the predicted pixel coordinates and the actual pixel coordinates; The plane estimation module is used to determine the plane equations of each undetermined plane according to the checkerboard corner points in each target image.

[0043] A third embodiment of the present application provides an electronic device, including: at least one processor; and a memory communicatively connected to at least one of the processors; wherein, The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned multi-view oriented external parameter calibration and plane estimation method.

[0044] The fourth embodiment of the present application provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to be executed by the computer to implement the above-mentioned multi-view extrinsic parameter calibration and plane estimation method.

[0045] Reference below Figure 2 , which shows a schematic diagram of the structure of a computer system of a server suitable for implementing the method and system embodiments of the present application. Figure 2 The server shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0046] like Figure 2 As shown, the computer system includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 302 or the program loaded from the storage part 308 to the random access memory (RAM) 303. Various programs and data required for system operation are also stored in the RAM 303. The CPU 301, ROM 302 and RAM 303 are connected to each other through a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0047] The following components are connected to the I / O interface 305: an input section 306 including a keyboard, a mouse, etc.; an output section 307 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 303 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed so that a computer program read therefrom is installed into the storage section 308 as needed.

[0048] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 309, and / or installed from the removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, the above-mentioned functions defined in the method of the present application are executed. It should be noted that the above-mentioned computer-readable medium of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, - but not limited to - a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above.

[0049] More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, an apparatus or a device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, an apparatus or a device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0050] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0051] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0052] The terms "first", "second", etc. are used to distinguish similar objects rather than to describe or indicate a particular order or sequence.

[0053] The term "comprise" or any other similar term is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus / device that includes a list of elements includes not only those elements but also other elements not expressly listed, or also includes elements inherent to such process, method, article, or apparatus / device.

[0054] So far, the technical solution of the present invention has been described in conjunction with the preferred embodiments shown in the drawings.

[0055] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A multi-view extrinsic parameter calibration and plane estimation method, characterized in that: The method comprises: Acquire a target image, wherein the target image is obtained by sequentially capturing images of a plurality of undetermined planes on which checkerboards are placed, using a target camera based on a plurality of preset camera positions; Preprocessing the target image and extracting checkerboard corner points from each preprocessed target image, wherein the preprocessing includes graying, filtering and edge detection preprocessing; According to the checkerboard corner points in each target image, a target equation group of each target image is jointly established, and a preset optimization algorithm is used to solve the target equation group to obtain the external parameters of the target camera, wherein the target equation group is constructed based on the mapping relationship between the world coordinate system and the camera coordinate system where the checkerboard corner points in each target image are located, and the preset optimization algorithm solves the minimum value of a preset objective function by iteratively updating algorithm parameters to optimize the external parameters of the target camera, and the objective function is used to indicate the residual between the predicted pixel coordinates and the actual pixel coordinates; According to the checkerboard corner points in each target image, the plane equations of each undetermined plane are determined.

2. The method for multi-view extrinsic calibration and plane estimation according to claim 1, characterized in that: The method of combining a set of equations of multiple target images according to the checkerboard corner points in each target image and solving the target set of equations using a preset optimization algorithm to obtain the external parameters of the target camera includes: Determine the pixel coordinates and world coordinates of the checkerboard corner points in each target image; According to the pixel coordinates and world coordinates of the corner points of the chessboard, a target equation group after the multi-graph combination is determined, and the target equation group is solved jointly to obtain initial external parameters; The initial extrinsic parameters are optimized using the Levenberg-Marquardt least squares optimization algorithm to obtain the extrinsic parameters of the target camera.

3. The multi-view extrinsic calibration and plane estimation method according to claim 2, characterized in that: The pixel coordinates and world coordinates of the checkerboard corner points in each target image are expressed in homogeneous coordinates as follows: ； The target equation group after the multiple graphs are combined is: ； In the formula, is the parameter to be solved, is the external parameter of the target camera.

4. The method for multi-view extrinsic calibration and plane estimation according to claim 2, characterized in that: When the pixel coordinates of the checkerboard corner points in multiple target images and the corresponding world coordinates are combined, the contribution value of each target image to the combined result is balanced by setting the corresponding weight of each target image.

5. The method for multi-view extrinsic calibration and plane estimation according to claim 1, characterized in that: The method further comprises: Determine the reprojection error of each undetermined plane before and after the external parameter calibration respectively; According to the reprojection error, it is determined whether the extrinsic parameters of the target camera meet a preset accuracy threshold.

6. The multi-view extrinsic calibration and plane estimation method according to claim 1, characterized in that: Checkerboard corner points in any undetermined plane The world coordinates satisfy: ； Passing point The normal vector is , then point The plane equation of the undetermined plane is: ； in, , the normal vector It is calculated from the three non-collinear corner points in the chessboard where the plane is placed; The homogeneous coordinate form of the normal vector n is , then point The normal vector n satisfies: ； Assume that the checkerboard corner points in another undetermined plane are ,but The world coordinates satisfy: ； in, Represents the position transformation matrix of a set of checkerboard corner points in the world coordinate system; Passing point The normal vector satisfy: ； Assumed normal vector The homogeneous coordinate form of is: , then point The plane equation of the undetermined plane is: 。 7. The multi-view extrinsic parameter calibration and plane estimation method according to claim 1, characterized in that: The step of determining the plane equations of each undetermined plane according to the checkerboard corner points in each target image includes: Determine the coordinates of the checkerboard corner points in the world coordinate system as follows: ； In the formula, , is the coordinate of each point in the world coordinate system when the chessboard is placed on the kth plane; In the chessboard corner point coordinate collection A fitting plane, wherein the fitting plane corresponds to a set of plane normal vectors and offsets to be solved; exist Randomly select three non-collinear points in , to construct two direction vectors to solve the plane normal vector and offset corresponding to the fitting plane, and the solution process satisfies: ； In the formula, ; ; Points The x-coordinate, y-coordinate and z-coordinate of exist In the remaining point set, traverse each point in turn Distance from the fitting plane , and set the distance threshold to determine the Is it less than the distance threshold? is less than the distance threshold, the current point is determined is a point in the plane; After the traversal is completed, The set with the largest number of interior points is determined to optimize the parameters of the fitting plane until the final plane equation is obtained.

8. A multi-view extrinsic calibration and plane estimation system, characterized in that: The system comprises: A data acquisition module is used to acquire a target image, wherein the target image is acquired by sequentially acquiring images of a plurality of undetermined planes on which checkerboards are placed, using a target camera based on a plurality of preset camera positions; A preprocessing module, used for preprocessing the target image and extracting checkerboard corner points in each preprocessed target image, wherein the preprocessing includes graying, filtering and edge detection preprocessing; An extrinsic parameter solving module is used to jointly establish a target equation group of each target image according to the checkerboard corner points in each target image, and solve the target equation group using a preset optimization algorithm to obtain the extrinsic parameters of the target camera, wherein the target equation group is constructed based on the mapping relationship between the world coordinate system and the camera coordinate system where the checkerboard corner points in each target image are located, and the preset optimization algorithm solves the minimum value of a preset objective function by iteratively updating algorithm parameters to optimize the extrinsic parameters of the target camera, and the objective function is used to indicate the residual between the predicted pixel coordinates and the actual pixel coordinates; The plane estimation module is used to determine the plane equations of each undetermined plane according to the checkerboard corner points in each target image.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to at least one of the processors; wherein, The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the multi-view extrinsic parameter calibration and plane estimation method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to be executed by the computer to implement the multi-view extrinsic parameter calibration and plane estimation method described in any one of claims 1-7.

Images