External parameter calibration method and device, computer equipment, computer readable storage medium and computer program product

Abstract

The invention relates to an external parameter calibration method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps that a first calibration image of a T-shaped light-emitting calibration object at a first visual angle and a second calibration image of the T-shaped light-emitting calibration object at a second visual angle are acquired, and spherical light-emitting objects are arranged on the two sides of the T-shaped light-emitting calibration object respectively; determining a first feature point location where a light emitting object is located from the first calibration image, and determining a second feature point location where the light emitting object is located from the second calibration image; determining a first sorting result corresponding to the first feature point location and a second sorting result corresponding to the second feature point location; establishing a matching relationship between the first feature point location and the second feature point location according to the first sorting result and the second sorting result; and generating a calibration external parameter of the first view angle relative to the second view angle by adopting the first feature point location and the second feature point location which have the matching relationship. By adopting the method, the external parameter calibration precision under different distance angles can be improved.

Description

Technical Field

[0001] The present application relates to the field of camera calibration technology, and in particular to an extrinsic parameter calibration method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] In a multi-camera system, accurate camera calibration is the basis for ensuring the correct fusion and processing of image data. The calibration of camera extrinsics is the basis for describing the relative postures between different cameras in a multi-camera system. In the traditional method, multiple cameras are used to shoot chessboard calibration object images from different perspectives, and the camera extrinsics are calibrated based on the feature point data in the chessboard calibration object images.

[0003] However, since the checkerboard calibration object is only applicable within a specific viewing angle range, when the above-mentioned checkerboard calibration method is still used in application scenarios beyond this viewing angle range, it is easy to have the problem of low external parameter calibration accuracy due to the inability to accurately identify the feature points of the checkerboard calibration object. Summary of the invention

[0004] Based on this, it is necessary to provide a high-precision external parameter calibration method, device, computer equipment, computer-readable storage medium and computer program product to address the above technical problems.

[0005] In a first aspect, the present application provides an external parameter calibration method, comprising:

[0006] Acquire a first calibration image for a T-shaped luminous calibration object acquired at a first viewing angle, and a second calibration image for the T-shaped luminous calibration object acquired at a second viewing angle, wherein the first viewing angle is different from the second viewing angle, and spherical luminous objects are respectively arranged on both sides of the T-shaped luminous calibration object;

[0007] Determine a first characteristic point position where each of the luminous objects is located from the first calibration image, and determine a second characteristic point position where each of the luminous objects is located from the second calibration image;

[0008] Determine a first sorting result corresponding to a first characteristic point position of each of the luminous objects, and a second sorting result corresponding to a second characteristic point position of each of the luminous objects;

[0009] According to the first sorting result and the second sorting result, establishing a matching relationship between the first feature point and the second feature point;

[0010] A first feature point and a second feature point that are in a matching relationship are used to generate a calibration extrinsic parameter of the first viewing angle relative to the second viewing angle.

[0011] In one embodiment, after establishing a matching relationship between the first feature point and the second feature point according to the first sorting result and the second sorting result, the method further includes:

[0012] Mapping each of the first feature points to a world coordinate system to obtain a first three-dimensional distance between the first feature points in the world coordinate system, and matching the first three-dimensional distance with a physical distance between the luminous objects;

[0013] Mapping each of the second feature points to the world coordinate system to obtain a second three-dimensional distance between the second feature points in the world coordinate system, and matching the second three-dimensional distance with the physical distance;

[0014] It is determined that both the first three-dimensional distance and the second three-dimensional distance match the physical distance.

[0015] In one embodiment, determining a first sorting result corresponding to a first characteristic point position of each of the luminous objects and a second sorting result corresponding to a second characteristic point position of each of the luminous objects includes:

[0016] Acquire a first vertical position of each of the first feature points in the first calibration image, and determine a first sorting result corresponding to each of the first feature points based on the first vertical position;

[0017] A second vertical position of each second feature point in the second calibration image is obtained, and a second sorting result corresponding to each second feature point is determined based on the second vertical position.

[0018] In one embodiment, the step of using the first feature point and the second feature point that are in a matching relationship to generate a calibration extrinsic parameter of the first viewing angle relative to the second viewing angle includes:

[0019] Using the first feature point and the second feature point that have a matching relationship, constructing a basic matrix of the first viewing angle relative to the second viewing angle;

[0020] Decomposing the basic matrix to obtain a rotation matrix and a translation matrix of the first viewing angle relative to the second viewing angle;

[0021] The rotation matrix and the translation matrix are used as the calibration extrinsic parameters.

[0022] In one of the embodiments, before decomposing the basic matrix to obtain the rotation matrix and translation matrix of the first perspective relative to the second perspective, the method further includes:

[0023] Processing the first feature points using the basic matrix to obtain feature projection points corresponding to each of the first feature points at the second viewing angle;

[0024] Drawing epipolar lines in the second calibration image using the feature projection points;

[0025] It is determined that the second feature point matches the epipolar line.

[0026] In one of the embodiments, the first viewing angle and the second viewing angle are viewing angles of any two cameras selected from a multi-camera system;

[0027] The method further comprises:

[0028] constructing an initial spanning tree corresponding to the multi-camera system using a calibration extrinsic parameter of the first viewing angle relative to the second viewing angle;

[0029] Randomly select a number of groups of calibrated external parameters of the first perspective relative to the second perspective from the initial spanning tree as sample data, and determine initial model parameters corresponding to the initial spanning tree;

[0030] generating predicted calibration extrinsic parameters between any two cameras in the multi-camera system using the initial model parameters, and determining an error between the predicted calibration extrinsic parameters and the calibration extrinsic parameters between the any two cameras;

[0031] Iteratively adjusting the initial model parameters according to the errors until the adjusted initial model parameters meet preset conditions, thereby obtaining adjusted model parameters;

[0032] The adjusted model parameters are used to update the initial spanning tree to obtain a target spanning tree, where the target spanning tree is used to characterize the connection relationship between the viewing angles of two cameras in the multi-camera system.

[0033] In a second aspect, the present application also provides an external parameter calibration device, comprising:

[0034] An image acquisition module, used to acquire a first calibration image of a T-shaped luminous calibration object acquired at a first viewing angle, and a second calibration image of the T-shaped luminous calibration object acquired at a second viewing angle, wherein the first viewing angle is different from the second viewing angle, and spherical luminous objects are respectively arranged on both sides of the T-shaped luminous calibration object;

[0035] a point determination module, used to determine a first characteristic point position where each of the luminous objects is located from the first calibration image, and to determine a second characteristic point position where each of the luminous objects is located from the second calibration image;

[0036] A point sorting module, used to determine a first sorting result corresponding to a first characteristic point of each of the luminous objects, and a second sorting result corresponding to a second characteristic point of each of the luminous objects;

[0037] A point matching module, used for establishing a matching relationship between the first feature point and the second feature point according to the first sorting result and the second sorting result;

[0038] The external parameter calibration module is used to generate a calibration external parameter of the first viewing angle relative to the second viewing angle by using a first feature point and a second feature point that have a matching relationship.

[0039] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the external parameter calibration method described in any one of the embodiments of the first aspect is implemented.

[0040] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the extrinsic parameter calibration method described in any one of the embodiments of the first aspect is implemented.

[0041] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the external parameter calibration method described in any one of the embodiments of the first aspect is implemented.

[0042] The above-mentioned external parameter calibration method, device, computer equipment, computer-readable storage medium and computer program product obtain calibration images for the T-shaped luminous calibration object collected at different viewing angles, match the feature points in the calibration image that match the luminous object of the T-shaped luminous calibration object according to the sorting results, and use the feature points with matching relationships to generate relative calibration external parameters between different viewing angles. The sorting results can be used to simplify the matching difficulty between feature points, reduce the possibility of mismatching, and thus improve the accuracy of external parameter calibration. In addition, because the T-shaped luminous calibration object is used for calibration in the above-mentioned external parameter calibration method, the acquisition of the calibration image can be free from the limitations of distance and viewing angle, thereby improving the applicability and flexibility of external parameter calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 An application environment diagram of an external parameter calibration method in an embodiment;

[0045] Figure 2 A schematic diagram of a flow chart of an external parameter calibration method in an embodiment;

[0046] Figure 3 A schematic diagram of a process flow of a geometric consistency verification step in one embodiment;

[0047] Figure 4 A schematic diagram of a flow chart of a calibration external parameter determination step in an embodiment;

[0048] Figure 5 is a schematic flow chart of an epipolar line correction step in one embodiment;

[0049] Figure 6 A schematic diagram of a global optimization process in one embodiment;

[0050] Figure 7 A schematic diagram of a flow chart of an external parameter calibration method in another embodiment;

[0051] Figure 8 is a structural block diagram of an external parameter calibration device 800 in one embodiment;

[0052] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0055] The external parameter calibration method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the camera 120 and the camera 122 can communicate with the server 160 through the network. The camera 120 can be used to collect a first calibration image of the T-shaped luminous calibration object 140 from a first viewing angle. The camera 122 can be used to collect a second calibration image of the T-shaped luminous calibration object 140 from a second viewing angle. The first viewing angle is different from the second viewing angle. Spherical luminous objects 142 are respectively arranged on both sides of the T-shaped luminous calibration object 140.

[0056] Specifically, the server 160 can obtain a first calibration image for the T-shaped luminous calibration object 140 captured by the camera 120 at a first viewing angle, and a second calibration image for the T-shaped luminous calibration object 140 captured by the camera 122 at a second viewing angle. The server 160 can determine the first feature point position where each luminous object 142 is located from the first calibration image, and determine the second feature point position where each luminous object 142 is located from the second calibration image. The server 160 can determine a first sorting result corresponding to the first feature point position of each luminous object 142, and a second sorting result corresponding to the second feature point position of each luminous object 142. The server 160 can establish a matching relationship between the first feature point position and the second feature point position based on the first sorting result and the second sorting result. The server 160 can use the first feature point position and the second feature point position that have a matching relationship to generate a calibration external parameter of the first viewing angle relative to the second viewing angle.

[0057] The server 160 may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0058] In an exemplary embodiment, Figure 2 As shown, an external parameter calibration method is provided, which is applied to Figure 1 The server 160 in the example is used as an example to illustrate the method, which includes the following steps S202 to S210. Among them:

[0059] Step S202: acquiring a first calibration image for the T-shaped luminous calibration object acquired at a first viewing angle, and a second calibration image for the T-shaped luminous calibration object acquired at a second viewing angle.

[0060] Among them, the first perspective is different from the second perspective. There is no difference in priority or angle between the first perspective and the second perspective. "First" and "second" are only used to distinguish different perspectives. Exemplarily, the image first captured can be used as the first calibration image under the first perspective, or the image first received by the server can be used as the first calibration image under the first perspective, and the other image is the second calibration image under the second perspective different from the first calibration image.

[0061] Spherical luminous objects are respectively arranged on both sides of the T-shaped luminous calibration object.

[0062] For example, the server can respond to the external parameter calibration request triggered by the user, and obtain the first calibration image of the T-shaped luminous calibration object captured at the first viewing angle and the second calibration image of the T-shaped luminous calibration object captured at the second viewing angle from cameras of different viewing angles. Alternatively, the server can also automatically trigger the external parameter calibration request after the camera is initialized and start, and obtain different calibration images captured by different cameras at different viewing angles.

[0063] Step S204: determining a first characteristic point position of each luminous object from the first calibration image, and determining a second characteristic point position of each luminous object from the second calibration image.

[0064] Exemplarily, the server can perform feature extraction on the first calibration image, match the extracted feature data with the shape feature or grayscale feature of the luminous object, and determine the pixel position corresponding to each luminous object in the T-shaped luminous calibration object in the first calibration image, that is, obtain the first feature point position. Similarly, feature extraction can also be performed on the second calibration image, and the second feature point position corresponding to the T-shaped luminous calibration object in the second calibration image can be determined by using the matching result of the extracted feature data and the luminous object.

[0065] Step S206, determining a first sorting result corresponding to the first characteristic point position of each luminous object and a second sorting result corresponding to the second characteristic point position of each luminous object.

[0066] For example, the server may sort the first feature points in the first calibration image based on the detection order, position or matching degree of each feature point with the luminous object, and determine the first sorting result corresponding to the first feature point of each luminous object. Similarly, the second sorting result corresponding to the second feature point of each luminous object is determined by using the same sorting method as the first sorting result.

[0067] Step S208: establishing a matching relationship between the first feature point and the second feature point according to the first sorting result and the second sorting result.

[0068] For example, the server may query the first feature point according to the first sorting result corresponding to the first feature point to obtain the second feature point whose second sorting result is the same as the first sorting result, and establish a matching relationship between the first feature point whose first sorting result is the same as the second sorting result and the second feature point.

[0069] Step S210: using the first feature point and the second feature point that have a matching relationship, generate a calibration extrinsic parameter of the first viewing angle relative to the second viewing angle.

[0070] Exemplarily, the server may use the second feature point with a matching relationship as the projection of the first feature point in the second calibration image at the second viewing angle. The second feature point with a matching relationship is used to perform calculation processing with the first feature point to obtain a transformation matrix from the imaging coordinate system where the first calibration image is located to the imaging coordinate system where the second calibration image is located. The intrinsic parameters of the camera that captured the first calibration image and the intrinsic parameters of the camera that captured the second calibration image are used to perform calculation processing with the transformation matrix to generate calibration extrinsic parameters of the first viewing angle relative to the second viewing angle.

[0071] In the above external parameter calibration method, by obtaining calibration images for the T-shaped luminous calibration object collected at different viewing angles, matching the feature points in the calibration image that match the luminous object of the T-shaped luminous calibration object according to the sorting results, and using the feature points with matching relationships to generate relative calibration external parameters between different viewing angles, the sorting results can be used to simplify the matching difficulty between feature points, reducing the possibility of mismatching, thereby improving the accuracy of external parameter calibration. In addition, because the T-shaped luminous calibration object is used for calibration in the above external parameter calibration method, the acquisition of the calibration image can be free from the limitations of distance and viewing angle, thereby improving the applicability and flexibility of external parameter calibration.

[0072] In an exemplary embodiment, Figure 3 As shown, a method for verifying the geometric consistency of the first feature point and the second feature point is also provided, including the following steps S302 to S306. Among them:

[0073] Step S302 , mapping each first feature point to a world coordinate system to obtain a first three-dimensional distance between the first feature points in the world coordinate system, and matching the first three-dimensional distance with a physical distance between the luminous objects.

[0074] Step S304: Map each second feature point to the world coordinate system to obtain a second three-dimensional distance between the second feature points in the world coordinate system, and match the second three-dimensional distance with the physical distance.

[0075] Exemplarily, the server can determine the luminous object corresponding to the first feature point based on the sorting result of the first feature point. According to the three-dimensional spatial coordinates of the luminous object in the world coordinate system and the two-dimensional image coordinates of the first feature point in the imaging coordinate system where the first calibration image is located, a transformation matrix between the imaging coordinate system where the first calibration image is located and the world coordinate system is established. The three-dimensional spatial coordinates corresponding to another first feature point in the world coordinate system are determined using the transformation matrix. The first three-dimensional distance between the first feature points in the world coordinate system is determined using the three-dimensional spatial coordinates corresponding to each first feature point in the world coordinate system. The first three-dimensional distance is matched with the actual physical distance between the two luminous objects in the T-shaped luminous calibration object.

[0076] Similarly, the server can refer to the first three-dimensional distance determination method in step S302 to process the second feature points to obtain the second three-dimensional distance between the second feature points in the world coordinate system, and match the second three-dimensional distance with the actual physical distance between the luminous objects.

[0077] Step S306: determine whether the first three-dimensional distance and the second three-dimensional distance both match the physical distance.

[0078] Exemplarily, when the first three-dimensional distance is equal to the physical distance, or the error between the first three-dimensional distance and the physical distance is less than a preset error threshold, the server can determine that the first three-dimensional distance matches the physical distance. Similarly, when the second three-dimensional distance is equal to the physical distance or the error is less than a preset error threshold, it can be determined that the second three-dimensional distance matches the physical distance.

[0079] In this embodiment, after determining the matching relationship, the first and second two-dimensional feature points are mapped to the world coordinate system to determine the first three-dimensional distance corresponding to the first feature point and the second three-dimensional distance corresponding to the second feature point. By matching the first and second three-dimensional distances with the actual physical distance between the luminous object, the geometric consistency of the feature points with a matching relationship can be verified in combination with the geometric features of the T-shaped luminous calibration object, thereby ensuring the accuracy of the feature points with a matching relationship.

[0080] In an exemplary embodiment, the above step S206 may also be implemented in the following manner:

[0081] The first vertical position of each first feature point in the first calibration image is obtained, and a first sorting result corresponding to each first feature point is determined based on the first vertical position. The second vertical position of each second feature point in the second calibration image is obtained, and a second sorting result corresponding to each second feature point is determined based on the second vertical position.

[0082] The vertical position can be used to represent the coordinate position in the vertical axis direction.

[0083] Exemplarily, when detecting the first feature point, the server may compare the first vertical position of each first feature point in the first calibration image. The first feature point with a higher first vertical position is determined as the first feature point with a sorting result of 1, and the first feature point with a first vertical position below the first feature point with a sorting result of 1 is determined as the first feature point with a sorting result of 2. Similarly, the second sorting result corresponding to each second feature point may be determined according to the height of the second vertical position.

[0084] Optionally, in some implementations, the server may further output the corresponding first feature points in sequence according to the first sorting result, and output the corresponding second feature points in sequence according to the second sorting result.

[0085] In this embodiment, by using the vertical position to determine the sorting results corresponding to the feature points, it is possible to ensure that the output order of the feature points when outputting each calibration image is consistent, thereby helping to simplify the matching process of the feature points.

[0086] In an exemplary embodiment, Figure 4 As shown, the above step S210 includes the following steps S402 to S406. Among them:

[0087] Step S402: construct a basic matrix of the first viewing angle relative to the second viewing angle by using the first feature point and the second feature point that have a matching relationship.

[0088] Exemplarily, the first feature point and the second feature point may include position information of a plurality of pixels corresponding to the luminous object. The server may determine a plurality of feature point pairs using the position information of a plurality of pixels in the first feature point and the second feature point that have a matching relationship. Seven feature point pairs, eight feature point pairs, or ten feature point pairs are used for computational processing to construct a basic matrix of the first perspective relative to the second perspective.

[0089] Step S404: decompose the basic matrix to obtain a rotation matrix and a translation matrix of the first viewing angle relative to the second viewing angle.

[0090] Step S406: Use the rotation matrix and the translation matrix as calibration extrinsic parameters.

[0091] Exemplarily, the server may use an intrinsic matrix decomposition algorithm to decompose the basic matrix and convert the basic matrix into the product of a rotation matrix and a translation matrix. Thus, the rotation matrix and the translation matrix of the first perspective relative to the second perspective are obtained. The rotation matrix and the translation matrix are used as calibration external parameters of the first perspective relative to the second perspective.

[0092] In this embodiment, a basic matrix of the first perspective relative to the second perspective is constructed by utilizing feature points that have a matching relationship, and the basic matrix is ​​decomposed to obtain a rotation matrix and a translation matrix. The rotation matrix and the translation matrix are used as calibration extrinsic parameters of the first perspective relative to the second perspective, which can improve the determination accuracy of the calibration extrinsic parameters.

[0093] In an exemplary embodiment, Figure 5 As shown, before the above step S404, the following steps S502 to S506 may also be included. Among them:

[0094] Step S502: Process the first feature points using the basic matrix to obtain feature projection points corresponding to each first feature point at a second viewing angle.

[0095] Step S504: draw epipolar lines in the second calibration image using the feature projection points.

[0096] Step S506: determine whether the second feature point matches the epipolar line.

[0097] Exemplarily, the server may multiply the first feature point by the basic matrix, and use the data result obtained after the multiplication as the feature projection point of the first feature point under the second viewing angle. The corresponding feature projection points are connected in sequence according to the sorting result of the first feature point to form an epipolar line corresponding to the first feature point in the second calibration image. The second feature point is matched with the epipolar line using the position of the second feature point and the position of the epipolar line. When the second feature point is included in the epipolar line, it is determined that the second feature point matches the epipolar line.

[0098] In this embodiment, by using the feature projection point of the first feature point to draw the epipolar line in the second calibration image, when the second feature point matches the epipolar line, it is determined that the first feature point and the second feature point match correctly, which can visualize the effect of the epipolar geometry and facilitate subsequent epipolar constraints, thereby helping to improve the accuracy of subsequent extrinsic parameter calibration.

[0099] In an exemplary embodiment, the first viewing angle and the second viewing angle are the viewing angles of any two cameras selected from a multi-camera system. Figure 6 As shown, the external parameter calibration method may further include steps S602 to S610. Among them:

[0100] Step S602: construct an initial spanning tree corresponding to the multi-camera system using calibrated extrinsic parameters of the first viewing angle relative to the second viewing angle.

[0101] Exemplarily, the server may adopt a maximum spanning tree algorithm, using each camera in the multi-camera system as a node in the spanning tree, and using calibrated extrinsic parameters between each pair of cameras as connection links between nodes, thereby constructing an initial spanning tree corresponding to the multi-camera system.

[0102] Step S604: randomly select several groups of calibrated external parameters of the first perspective relative to the second perspective from the initial spanning tree as sample data, and determine initial model parameters corresponding to the initial spanning tree.

[0103] For example, the server can use the connection links between two nodes in the initial spanning tree as a set of calibration external parameters of the first perspective relative to the second perspective, and randomly select several groups of data from the initial spanning tree as sample data. The sample data is processed by least squares method to fit the initial model parameters of the initial spanning tree.

[0104] Step S606, using the initial model parameters to generate predicted calibration extrinsic parameters between any two cameras in the multi-camera system, and determining an error between the predicted calibration extrinsic parameters and the calibration extrinsic parameters between any two cameras.

[0105] For example, the server can use the initial model parameters and the node calculation of any camera in the multi-camera system to obtain the connection link between the node and another node, thereby obtaining the predicted calibration extrinsic parameters between any two cameras. The error between the predicted calibration extrinsic parameters between the nodes predicted by the initial model parameters and the actual calibration extrinsic parameters in the initial spanning tree is calculated.

[0106] Step S608, iteratively adjusting the initial model parameters according to the errors until the adjusted initial model parameters meet preset conditions, thereby obtaining adjusted model parameters.

[0107] Exemplarily, the server may compare the error with a preset threshold. When the error is less than the preset threshold, the node corresponding to the predicted calibration external parameter is used as the node matching the initial model parameter. The number of nodes matching the initial model parameters in the initial spanning tree is determined. The initial model parameters are iteratively optimized until the number of nodes matched using the optimized model parameters reaches a preset number threshold, thereby obtaining the adjusted model parameters.

[0108] Step S610: The initial spanning tree is updated with the adjusted model parameters to obtain a target spanning tree, where the target spanning tree is used to characterize the connection relationship between the viewing angles of each two cameras in the multi-camera system.

[0109] Exemplarily, the server may use the adjusted model parameters to update the connection links between the nodes in the initial spanning tree, thereby obtaining the target spanning tree. The connection links between the nodes in the target spanning tree may be used to characterize the connection relationship between the viewing angles of the cameras in the multi-camera system.

[0110] In this embodiment, by using the maximum spanning tree and random sampling consistency algorithm to process the calibration extrinsic parameters between any two cameras in the multi-camera system, the global calibration consistency and calibration accuracy of the multi-camera system can be guaranteed.

[0111] In an exemplary embodiment, Figure 7 As shown, an external parameter calibration algorithm is also provided, including the following steps S702 to S718. Among them:

[0112] Step S702: acquiring a first calibration image for the T-shaped luminous calibration object acquired at a first viewing angle, and a second calibration image for the T-shaped luminous calibration object acquired at a second viewing angle.

[0113] Step S704: determine the first characteristic point position of each luminous object from the first calibration image, and determine the second characteristic point position of each luminous object from the second calibration image.

[0114] For example, the server may obtain a first calibration image and a second calibration image collected for a T-shaped luminous calibration object, perform target detection on the first calibration image and the second calibration feature, respectively, and determine a first feature point position of the luminous object in the first calibration image and a second feature point position of the luminous object in the second calibration image.

[0115] Step S706: determining a first sorting result corresponding to each first feature point according to the first vertical position, and determining a second sorting result corresponding to each second feature point according to the second vertical position.

[0116] For example, the server may respectively use the first vertical position of the first feature point to determine the first sorting result corresponding to the first feature point, and use the second vertical position of the second feature point to determine the second sorting result corresponding to the second feature point, and sequentially output the corresponding first feature points according to the first sorting result, and sequentially output the corresponding second feature points according to the second sorting result.

[0117] Step S708: establishing a matching relationship between the first feature point and the second feature point according to the first sorting result and the second sorting result.

[0118] Step S710: Determine that a first three-dimensional distance mapped from a first feature point to a world coordinate system and a second three-dimensional distance mapped from a second feature point to a world coordinate system both match a physical distance between luminous objects.

[0119] Exemplarily, the server may respectively establish a matching relationship between a first feature point whose first sorting result is 1 and a second feature point whose second sorting result is 1, and a matching relationship between a first feature point whose first sorting result is 2 and a second feature point whose second sorting result is 2. The first three-dimensional distance of the first feature point in the world coordinate system is determined by using the three-dimensional spatial coordinates of the luminous object in the world coordinate system and the first feature point corresponding to the luminous object. Similarly, the second three-dimensional distance of the second feature point in the world coordinate system is determined by using the three-dimensional spatial coordinates of the luminous object in the world coordinate system and the second feature point corresponding to the luminous object. The first three-dimensional distance and the second three-dimensional distance are matched with the actual physical distance of the luminous object in the world coordinate system respectively. When the error between the first three-dimensional distance and the physical distance is within the preset error threshold, and the error between the second three-dimensional distance and the physical distance is within the preset error threshold, it is determined that the matching relationship between the first feature point and the second feature point is established correctly.

[0120] Step S712: construct a basic matrix of the first viewing angle relative to the second viewing angle by using the first feature point and the second feature point that have a matching relationship.

[0121] Step S714: draw epipolar lines in the second calibration image using the basic matrix and the first feature points, and determine whether the second feature points match the epipolar lines.

[0122] Step S716, decomposing the basic matrix to obtain a rotation matrix and a translation matrix of the first perspective relative to the second perspective, and using the rotation matrix and the translation matrix as calibration extrinsic parameters.

[0123] Exemplarily, the server can use the eight-point algorithm to perform calculations on the first feature point and the second feature point that have a matching relationship, and construct a basic matrix of the first perspective relative to the second perspective. The basic matrix is ​​used to determine the feature projection point corresponding to the first feature point at the second perspective, and the polar line is drawn using the feature projection point. Determine that the second feature point matches the polar line. The basic matrix is ​​decomposed using the intrinsic matrix decomposition algorithm to obtain the rotation matrix and translation matrix of the first perspective relative to the second perspective, and the rotation matrix and translation matrix are used as calibration external parameters.

[0124] Step S718, constructing a target spanning tree corresponding to the multi-camera system using the calibrated extrinsic parameters of the first perspective relative to the second perspective.

[0125] Exemplarily, the server can construct an initial spanning tree using the calibrated extrinsic parameters between any two cameras in the multi-camera system. Randomly select several groups of calibrated extrinsic parameters of the first perspective relative to the second perspective from the initial spanning tree as sample data, and determine the initial model parameters corresponding to the initial spanning tree. Generate predicted calibrated extrinsic parameters between any two cameras in the multi-camera system using the initial model parameters, and determine the error between the predicted calibrated extrinsic parameters and the calibrated extrinsic parameters between any two cameras. Iteratively adjust the initial model parameters according to the error until the adjusted initial model parameters meet the preset conditions, and obtain the adjusted model parameters. Use the adjusted model parameters to update the initial spanning tree to obtain a target spanning tree, which is used to characterize the connection relationship between the perspectives of two cameras in the multi-camera system.

[0126] In this embodiment, by using a T-shaped luminous calibration object for external parameter calibration, the limitations of the traditional checkerboard calibration method in the calibration scene and calibration angle can be overcome, more spatial scene information can be provided, and the flexibility of the external parameter calibration method can be improved. By using an actively luminous luminous object to locate the feature points in the calibration image, the positioning efficiency and positioning accuracy of the feature points can be improved. By determining the sorting results of each feature point according to the vertical position and matching the feature points according to the sorting results, the matching process of the feature points can be simplified, the matching efficiency of the feature points can be improved, and the probability of mismatching can be reduced. By mapping the feature points to the world coordinate system and verifying the feature points using the actual physical distance of the luminous object, the geometric consistency of the matched feature points and the T-shaped luminous calibration object can be guaranteed. By using the basic matrix to construct the polar line to determine the calibration external parameters, the determination accuracy of the calibration external parameters can be improved. By optimizing the calibration parameters between the two cameras in the multi-camera system through the maximum spanning tree algorithm and the random sampling consistency algorithm, the global calibration consistency and calibration accuracy of the multi-camera system can be guaranteed.

[0127] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indications of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0128] Based on the same inventive concept, the embodiment of the present application also provides an external parameter calibration device for implementing the external parameter calibration method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more external parameter calibration device embodiments provided below can refer to the limitations of the external parameter calibration method above, and will not be repeated here.

[0129] In an exemplary embodiment, Figure 8 As shown, an external parameter calibration device 800 is provided, comprising: an image acquisition module 802, a point determination module 804, a point sorting module 806, a point matching module 808 and an external parameter calibration module 810, wherein:

[0130] The image acquisition module 802 is used to acquire a first calibration image of the T-shaped luminous calibration object captured at a first viewing angle, and a second calibration image of the T-shaped luminous calibration object captured at a second viewing angle. The first viewing angle is different from the second viewing angle, and spherical luminous objects are respectively arranged on both sides of the T-shaped luminous calibration object.

[0131] The point determination module 804 is used to determine the first characteristic point of each luminous object from the first calibration image, and determine the second characteristic point of each luminous object from the second calibration image.

[0132] The point sorting module 806 is used to determine a first sorting result corresponding to the first characteristic point of each luminous object and a second sorting result corresponding to the second characteristic point of each luminous object.

[0133] The point matching module 808 is used to establish a matching relationship between the first feature point and the second feature point according to the first sorting result and the second sorting result.

[0134] The external parameter calibration module 810 is used to generate a calibration external parameter of the first viewing angle relative to the second viewing angle by using the first feature point and the second feature point that have a matching relationship.

[0135] In an exemplary embodiment, the external parameter calibration device 800 also includes a geometric verification module, which is used to map each first feature point to the world coordinate system to obtain a first three-dimensional distance between the first feature points in the world coordinate system, and match the first three-dimensional distance with the physical distance between the luminous objects; map each second feature point to the world coordinate system to obtain a second three-dimensional distance between the second feature points in the world coordinate system, and match the second three-dimensional distance with the physical distance; and determine that both the first three-dimensional distance and the second three-dimensional distance match the physical distance.

[0136] In an exemplary embodiment, the point sorting module 806 is also used to obtain the first vertical position of each first feature point in the first calibration image, and determine a first sorting result corresponding to each first feature point based on the first vertical position; obtain the second vertical position of each second feature point in the second calibration image, and determine a second sorting result corresponding to each second feature point based on the second vertical position.

[0137] In an exemplary embodiment, the external parameter calibration module 810 is also used to use the first feature point and the second feature point that have a matching relationship to construct a basic matrix of the first perspective relative to the second perspective; decompose the basic matrix to obtain a rotation matrix and a translation matrix of the first perspective relative to the second perspective; and use the rotation matrix and the translation matrix as calibration external parameters.

[0138] In an exemplary embodiment, the external parameter calibration module 810 is also used to process the first feature points using the basic matrix to obtain feature projection points corresponding to each first feature point at a second viewing angle; draw epipolar lines in the second calibration image using the feature projection points; and determine that the second feature points match the epipolar lines.

[0139] In an exemplary embodiment, the first perspective and the second perspective are perspectives of any two cameras selected from the multi-camera system. The extrinsic parameter calibration device 800 also includes: a global optimization module for constructing an initial spanning tree corresponding to the multi-camera system using the calibration extrinsic parameters of the first perspective relative to the second perspective; randomly selecting a number of groups of calibration extrinsic parameters of the first perspective relative to the second perspective from the initial spanning tree as sample data, and determining initial model parameters corresponding to the initial spanning tree; using the initial model parameters to generate predicted calibration extrinsic parameters between any two cameras in the multi-camera system, and determining the error between the predicted calibration extrinsic parameters and the calibration extrinsic parameters between any two cameras; iteratively adjusting the initial model parameters according to the errors until the adjusted initial model parameters meet the preset conditions, and obtaining the adjusted model parameters; using the adjusted model parameters to update the initial spanning tree to obtain a target spanning tree, and the target spanning tree is used to characterize the connection relationship between the perspectives of two cameras in the multi-camera system.

[0140] Each module in the above-mentioned external parameter calibration device 800 can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0141] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig. 9As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as a first calibration image, a second calibration image, a first feature point, a second feature point, a first sorting result, a second sorting result, a matching relationship, and a calibration external parameter. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an external parameter calibration method is implemented.

[0142] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0143] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0144] In an exemplary embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0145] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0146] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0147] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0148] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for calibrating an external parameter, characterized in that: The method comprises: Acquire a first calibration image for a T-shaped luminous calibration object acquired at a first viewing angle, and a second calibration image for the T-shaped luminous calibration object acquired at a second viewing angle, wherein the first viewing angle is different from the second viewing angle, and spherical luminous objects are respectively arranged on both sides of the T-shaped luminous calibration object; Determine a first characteristic point position where each of the luminous objects is located from the first calibration image, and determine a second characteristic point position where each of the luminous objects is located from the second calibration image; Determine a first sorting result corresponding to a first characteristic point position of each of the luminous objects, and a second sorting result corresponding to a second characteristic point position of each of the luminous objects; According to the first sorting result and the second sorting result, establishing a matching relationship between the first feature point and the second feature point; A first feature point and a second feature point that are in a matching relationship are used to generate a calibration extrinsic parameter of the first viewing angle relative to the second viewing angle.

2. The method according to claim 1, characterized in that After establishing a matching relationship between the first feature point and the second feature point according to the first sorting result and the second sorting result, the method further includes: Mapping each of the first feature points to a world coordinate system to obtain a first three-dimensional distance between the first feature points in the world coordinate system, and matching the first three-dimensional distance with a physical distance between the luminous objects; Mapping each of the second feature points to the world coordinate system to obtain a second three-dimensional distance between the second feature points in the world coordinate system, and matching the second three-dimensional distance with the physical distance; It is determined that both the first three-dimensional distance and the second three-dimensional distance match the physical distance.

3. The method according to claim 1, characterized in that The determining of a first sorting result corresponding to a first characteristic point position of each of the luminous objects and a second sorting result corresponding to a second characteristic point position of each of the luminous objects includes: Acquire a first vertical position of each of the first feature points in the first calibration image, and determine a first sorting result corresponding to each of the first feature points based on the first vertical position; A second vertical position of each second feature point in the second calibration image is obtained, and a second sorting result corresponding to each second feature point is determined based on the second vertical position.

4. The method according to claim 1, characterized in that: The step of using the first feature point and the second feature point that have a matching relationship to generate a calibration extrinsic parameter of the first viewing angle relative to the second viewing angle includes: Using the first feature point and the second feature point that have a matching relationship, constructing a basic matrix of the first viewing angle relative to the second viewing angle; Decomposing the basic matrix to obtain a rotation matrix and a translation matrix of the first viewing angle relative to the second viewing angle; The rotation matrix and the translation matrix are used as the calibration extrinsic parameters.

5. The method according to claim 4, characterized in that Before decomposing the basic matrix to obtain a rotation matrix and a translation matrix of the first viewing angle relative to the second viewing angle, the method further includes: Processing the first feature points using the basic matrix to obtain feature projection points corresponding to each of the first feature points at the second viewing angle; Drawing epipolar lines in the second calibration image using the feature projection points; It is determined that the second feature point matches the epipolar line.

6. The method according to claim 1, characterized in that The first viewing angle and the second viewing angle are viewing angles of any two cameras selected from a multi-camera system; The method further comprises: constructing an initial spanning tree corresponding to the multi-camera system using a calibration extrinsic parameter of the first viewing angle relative to the second viewing angle; Randomly select a number of groups of calibrated external parameters of the first perspective relative to the second perspective from the initial spanning tree as sample data, and determine initial model parameters corresponding to the initial spanning tree; generating predicted calibration extrinsic parameters between any two cameras in the multi-camera system using the initial model parameters, and determining an error between the predicted calibration extrinsic parameters and the calibration extrinsic parameters between the any two cameras; Iteratively adjusting the initial model parameters according to the errors until the adjusted initial model parameters meet preset conditions, thereby obtaining adjusted model parameters; The adjusted model parameters are used to update the initial spanning tree to obtain a target spanning tree, where the target spanning tree is used to characterize the connection relationship between the viewing angles of two cameras in the multi-camera system.

7. An external parameter calibration device, characterized in that: The device comprises: An image acquisition module, used to acquire a first calibration image of a T-shaped luminous calibration object acquired at a first viewing angle, and a second calibration image of the T-shaped luminous calibration object acquired at a second viewing angle, wherein the first viewing angle is different from the second viewing angle, and spherical luminous objects are respectively arranged on both sides of the T-shaped luminous calibration object; a point determination module, used to determine a first characteristic point position where each of the luminous objects is located from the first calibration image, and to determine a second characteristic point position where each of the luminous objects is located from the second calibration image; A point sorting module, used to determine a first sorting result corresponding to a first characteristic point of each of the luminous objects, and a second sorting result corresponding to a second characteristic point of each of the luminous objects; A point matching module, used for establishing a matching relationship between the first feature point and the second feature point according to the first sorting result and the second sorting result; The external parameter calibration module is used to generate a calibration external parameter of the first viewing angle relative to the second viewing angle by using a first feature point and a second feature point that have a matching relationship.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images