Method and device for external parameter calibration and three-dimensional reconstruction of multiple cameras and medium

Abstract

The invention provides external parameter calibration and three-dimensional reconstruction methods and equipment for multiple cameras, and a medium. The method comprises the following steps: acquiring a picture shot by each camera in the multiple cameras and an internal parameter of each camera; the picture comprises at least part of preset feature points of the calibration plate, and the calibration plate is a rotation invariant calibration plate; identifying preset feature points in the pictures, and taking the pictures comprising the same preset feature points as a picture combination; and according to the same preset feature point and the internal reference, determining a position relationship between the cameras corresponding to the picture combination, and according to the position relationship, calculating external parameters of the plurality of cameras. According to the embodiment of the invention, each camera can be utilized to take a picture, the internal reference of the camera is combined, the external reference calibration of a plurality of cameras is simultaneously carried out, the operation is simple, the calibration process is efficient, and the result is more accurate.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, device and medium for extrinsic calibration and three-dimensional reconstruction of multiple cameras. Background Art

[0002] The camera's extrinsic parameters describe the camera's position and orientation in the world coordinate system and can be used to transform a 3D point in the world coordinate system into the camera coordinate system. In a multi-camera calibration scenario, the existing solution makes the calibration plate appear in the common field of view of multiple cameras at the same time, uses the camera to take multiple photos of the calibration plate, and obtains the extrinsic parameters of each camera based on the pictures taken by each camera. This solution is complex to operate, and the calibration results are unstable and have strong randomness. Summary of the invention

[0003] The embodiments of the present application provide a method, device and medium for extrinsic calibration and three-dimensional reconstruction of multiple cameras to solve one or more of the above-mentioned technical problems.

[0004] In a first aspect, an embodiment of the present application provides a method for calibrating the external parameters of multiple cameras, including: obtaining a picture taken by each of the multiple cameras and the internal parameters of each camera; wherein the picture includes at least part of the preset feature points of a calibration plate, and the calibration plate is a rotationally invariant calibration plate; identifying the preset feature points in the picture, and taking the pictures including the same preset feature points as a picture combination; determining the positional relationship between the cameras corresponding to the picture combination based on the same preset feature points and the internal parameters; and calculating the external parameters of the multiple cameras based on the positional relationship.

[0005] In a second aspect, an embodiment of the present application provides a three-dimensional reconstruction method, including: obtaining external parameters of multiple cameras; the external parameters are obtained according to the above method; using the external parameters to process the images taken by the multiple cameras to obtain a three-dimensional reconstruction result.

[0006] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the above methods when executing the computer program.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of the above is implemented.

[0008] Compared with the related art, this application has the following advantages:

[0009] The present application provides an extrinsic parameter calibration method, a three-dimensional reconstruction method, a device and a medium for multiple cameras. The extrinsic parameter calibration method for multiple cameras first obtains a picture taken by each camera in the multiple cameras and an intrinsic parameter of each camera; wherein the picture includes at least part of the preset feature points of a calibration plate, and the calibration plate is a rotationally invariant calibration plate; thereby, each camera can take only one picture, and the pictures taken by all cameras can include all the preset feature points in the calibration plate, then the preset feature points in the pictures are identified, and the pictures including the same preset feature points are used as a picture combination to realize fusion processing of the feature point data in the pictures, and finally, according to the same preset feature points and the intrinsic parameters, the positional relationship between the cameras corresponding to the picture combination is determined, and according to the positional relationship, the extrinsic parameters of the multiple cameras are calculated.

[0010] This application can use each camera to take a photo, and perform image matching processing based on the photos taken by each of the multiple cameras to obtain an image combination, so as to combine the camera internal parameters to determine the positional relationship between the cameras corresponding to the image combination, and perform external parameter calibration of multiple cameras at the same time. The operation is simple, the calibration process is efficient, and the results are more accurate.

[0011] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments according to the present application and should not be regarded as limiting the scope of the present application.

[0013] Figure 1 A flow chart of an external parameter calibration method for multiple cameras provided in an embodiment of the present application is shown;

[0014] Figure 2 A top view of the arrangement of the camera and the calibration plate provided in an embodiment of the present application is shown;

[0015] Figure 3 A schematic diagram of a rotationally invariant calibration plate provided in an embodiment of the present application is shown;

[0016] Figure 4 A flowchart of an implementation of an external parameter calibration method for multiple cameras provided in an embodiment of the present application is shown;

[0017] Figure 5A schematic diagram of a scenario provided in an embodiment of the present application is shown;

[0018] Figure 6 A flow chart of a three-dimensional reconstruction method provided in an embodiment of the present application is shown;

[0019] Figure 7 A structural block diagram of an external parameter calibration device for multiple cameras provided in an embodiment of the present application is shown;

[0020] Figure 8 A structural block diagram of a three-dimensional reconstruction device provided in an embodiment of the present application is shown;

[0021] Fig. 9 A block diagram of an electronic device used to implement an embodiment of the present application is shown. DETAILED DESCRIPTION

[0022] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the concept or scope of the present application. Therefore, the drawings and descriptions are considered to be exemplary in nature and not restrictive.

[0023] To facilitate understanding of the technical solutions of the embodiments of the present application, the following describes the related technologies of the embodiments of the present application. The following related technologies can be combined with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application.

[0024] First, the basic knowledge of mathematical projection transformation involved in this scheme is introduced.

[0025]

[0026] In the above formula, f x 、f y is the focal length of the camera, in pixels; u0 and v0 are the principal points in the x and y directions; f x 、f y , u0, v0 constitute the internal parameter matrix. c , Y c , Z c is a 3D point in the camera coordinate system, and u and v are pixel coordinates in the pixel coordinate system.

[0027] According to formula (1), the 3D point in the camera coordinate system can be mapped to the pixel coordinate system based on the camera intrinsic parameters.

[0028] From equations (2) and (3), the depth value and pixel coordinates are known. According to the camera intrinsic parameters, the depth map can be converted to the corresponding camera coordinate system.

[0029] For scenarios where multiple cameras need to be calibrated with external parameters, the existing solution requires taking multiple photos for each camera and calibrating the camera based on the photos taken by each camera. This process has poor operability and low degree of automation and requires human participation. The results obtained have poor stability and strong randomness, and the ability to resist noise is poor.

[0030] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The several specific embodiments listed can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0031] The present application embodiment provides a method for calibrating the external parameters of multiple cameras. Figure 1 The flowchart of the extrinsic calibration method of multiple cameras according to an embodiment of the present application is shown. The method may include:

[0032] Step 101, obtaining a picture taken by each camera of a plurality of cameras and an internal parameter of each camera; wherein the picture includes at least part of preset feature points of a calibration plate, and the calibration plate is a rotationally invariant calibration plate.

[0033] In an embodiment of the present application, multiple cameras are set according to a determined relative position relationship. For example, multiple cameras can be arranged in sequence according to a straight line, an arc, or a circle. Use the camera that has completed the position setting to take a photo of the calibration plate. It should be noted that the calibration plate is a rotationally invariant calibration plate, and each camera can take a partial area of ​​the calibration plate. In the case where each camera takes a photo, the photos taken by all cameras that have completed the position setting may include all preset feature points in the calibration plate.

[0034] It should be noted that the calibration plate may be a combination of one or more calibration plates. In the case where the calibration plate is a combination of multiple calibration plates, the multiple calibration plates may not be arranged in the same plane. The preset feature points in the calibration plate are multiple points pre-selected in the calibration plate. The positional relationship between the multiple preset feature points is known. For example, in the case where the multiple preset feature points may be points located on the same line or the same plane, the distance between the feature points is known.

[0035] It should also be noted that the rotation-invariant calibration plate is a tool designed for camera calibration, which is characterized by the fact that feature points can be accurately identified and matched no matter how the image is rotated. This type of calibration plate is often used for camera calibration tasks in computer vision and image processing to ensure that images taken by the camera in different postures can be accurately calibrated. The characteristics of the rotation-invariant calibration plate include: symmetry, the rotation-invariant calibration plate usually has a highly symmetrical pattern, such as a circular dot matrix, a ring mark, or a checkerboard with a symmetrical structure. This symmetry ensures that the relative position of the feature points remains unchanged after any angle rotation in the image; uniqueness, each feature point or mark on the calibration plate has a unique identifier, so that the feature point can still be uniquely identified even after rotation. For example, use marks of different sizes or colors, or add a unique number next to the mark; high contrast, the pattern on the calibration plate usually has high contrast so that the feature points can be clearly detected under various lighting conditions. This is usually achieved by using black and white contrast or color contrast.

[0036] In this step, after the relative positions of multiple cameras are determined, the pictures taken by each of the multiple cameras and the internal parameters of each camera are obtained, so as to use the pictures and internal parameters of each camera to perform subsequent external parameter calibration. This process only requires taking one picture with each camera, and the operation is simple. By using a calibration plate with rotation invariance, the preset feature points included in the picture can be easily and efficiently identified. Each picture includes at least some of the preset feature points of the calibration plate, so that data analysis is performed on the feature points in the picture to obtain the external parameters of the camera.

[0037] Step 102, identifying preset feature points in the pictures, and combining pictures including the same preset feature points.

[0038] In the embodiment of the present application, each image includes at least some of the preset feature points of all the preset feature points of the calibration plate. The preset feature points in the image can be identified by a feature point detection method, wherein the feature point detection method includes but is not limited to one or more of the following methods: corner point detection (corner points are points in the image with large grayscale value changes, usually located at the edge or intersection of the object), scale-invariant feature transformation (with scale invariance and rotation invariance), accelerated robust features (SURF), fast feature points (FAST), and ORB (Oriented FAST and Rotated BRIEF).

[0039] Each picture includes at least some of the preset feature points of all the preset feature points of the calibration plate. For example, the calibration plate includes 100 feature points with serial numbers of 1-100, and the preset feature points included in the picture are identified. For example, picture 1 includes preset feature points with serial numbers of 1-15, picture 2 includes preset feature points with serial numbers of 5-30, and picture 3 includes preset feature points with serial numbers of 25-50, and the preset feature points in each picture are identified. Based on these preset feature points, the pictures are analyzed and processed. Both picture 1 and picture 2 contain preset feature points with serial numbers of 5-15, that is, picture 1 and picture 2 include the same preset feature points, so picture 1 and picture 2 are used as a picture combination. Similarly, picture 2 and picture 3 are used as a picture combination. It should be noted that the number of pictures in the picture combination can include two or more, which can be set according to actual needs.

[0040] In this step, the pictures are matched by identifying preset feature points in the pictures, and a picture combination including the same preset feature points is obtained after matching, so that the relationship between the cameras corresponding to the picture combination can be determined more accurately.

[0041] Step 103: determining the positional relationship between the cameras corresponding to the picture combination according to the same preset feature points and the internal parameters, and calculating the external parameters of the multiple cameras according to the positional relationship.

[0042] In an embodiment of the present application, for each picture in the picture combination, an external parameter is calculated based on the internal parameter of the corresponding camera with the same preset feature point, and the obtained external parameter can be used to describe the positional relationship between the cameras corresponding to the pictures in the picture combination. For example, the first picture combination includes pictures 1 and 2, and the second picture combination includes pictures 2 and 3. After determining the positional relationship between the camera of picture 1 and the camera of picture 2, the positional relationship between the camera of picture 2 and the camera of picture 3 is determined, and then the positional relationship between the camera of picture 1 and the camera of picture 3 can be determined based on the first two positional relationships.

[0043] After knowing the positional relationship between any two cameras among the multiple cameras, based on these positional relationships, the position and orientation of the camera in the world coordinate system can be determined, that is, the extrinsic parameters of the multiple cameras are calculated according to the camera extrinsic parameters corresponding to the picture combination. The obtained camera extrinsic parameters can be applied to relative pose estimation, picture registration and stitching, and some computer vision algorithms, such as feature matching and optical flow estimation, which mainly focus on the geometric relationship in the image and the relative motion between pixels without the need for actual physical scale.

[0044] In this step, based on the positional relationship between the cameras corresponding to the pictures in the picture combination, the positional relationship between any two cameras among the multiple cameras is determined, and then the camera's extrinsic parameters are calculated. This process does not require human intervention and can quickly and accurately obtain the camera's extrinsic parameters.

[0045] The present application provides a method for calibrating the external parameters of multiple cameras. The method first obtains the pictures taken by each camera in the multiple cameras and the internal parameters of each camera; wherein the pictures include at least part of the preset feature points of the calibration plate, and the calibration plate is a rotationally invariant calibration plate; thus, each camera can only take one picture, and the pictures taken by all cameras can include all the preset feature points in the calibration plate, and then the preset feature points in the pictures are identified, and the pictures including the same preset feature points are used as picture combinations to realize the fusion processing of the feature point data in the pictures, and finally, according to the same preset feature points and the internal parameters, the positional relationship between the cameras corresponding to the picture combination is determined, and the external parameters of the multiple cameras are calculated according to the positional relationship. The present application can use each camera to take a picture, and perform picture matching processing based on the pictures taken by each camera in the multiple cameras to obtain a picture combination, thereby combining the camera internal parameters to determine the positional relationship between the cameras corresponding to the picture combination, and simultaneously perform external parameter calibration of multiple cameras, which is simple to operate, efficient in the calibration process, and more accurate in results.

[0046] In a possible implementation, the identifying of preset feature points in the pictures and taking the pictures including the same preset feature points as a picture combination can be performed according to the following steps: determining the first point data of the pictures in the picture combination according to the same preset feature points; wherein the first point data is located in the three-dimensional coordinate system of the camera.

[0047] In the embodiment of the present application, after matching and grouping the pictures taken by multiple cameras, each picture combination corresponds to a set of the same preset feature points, and the same preset feature points are transformed in dimension to obtain the first point data of each picture in the picture combination, that is, to obtain the three-dimensional point data in the camera's three-dimensional coordinate system. The three-dimensional point in the camera's three-dimensional coordinate system is used to describe the position and direction of the object in the picture taken by the camera relative to the camera.

[0048] In this possible implementation, a three-dimensional point corresponding to the same preset feature point, that is, first point data, can be obtained, providing data support for a more accurate camera extrinsic calibration result.

[0049] In a possible implementation, determining the positional relationship between the cameras corresponding to the picture combination based on the same preset feature points and the internal parameters can be performed according to the following steps: determining the positional relationship between the cameras corresponding to the picture combination based on the same preset feature points, the first point data and the internal parameters.

[0050] In this possible implementation, after obtaining the same preset feature points, and determining the first point data of the pictures in the picture combination based on the same preset feature points, the positional relationship between the corresponding cameras of the picture combination is determined by combining the same preset feature points, the first point data and the internal parameters.

[0051] In this possible implementation, the positional relationship between the cameras corresponding to two or more photos in the picture combination can be determined, so as to determine the positional relationship between different cameras based on different combinations. In the process of determining the positional relationship between the cameras, the three-dimensional first point data is applied to improve the accuracy of the obtained positional relationship.

[0052] In a possible implementation, the identifying of preset feature points in the image can be performed according to the following steps: obtaining key point information of the image taken by the target camera; wherein the key point information is integer data; based on the key point information, identifying target key points in the image taken by the target camera that are the same as the preset feature points in the calibration plate; and using the target key points as preset feature points in the image taken by the target camera.

[0053] In this possible implementation, the key point information of the image taken by the target camera can be obtained through AprilTag (a visual tag system). The key point information of the image taken by the target camera can also be obtained through Aruco (a visual tag system), thereby obtaining the key point information of integer data.

[0054] It should be noted that each AprilTag contains a unique ID, which allows the system to identify and distinguish different tags. This ID is usually encoded in the black and white checkered pattern of the tag. AprilTag consists of black and white squares, and this high-contrast design allows the tag to be reliably detected and identified under various lighting conditions. The four corner points of AprilTag are key points, which are used to calculate the position and posture of the tag in the image. By detecting these corner points, the translation and rotation of the tag relative to the camera can be estimated. The AprilTag system is highly robust and can still reliably detect and identify tags in the presence of noise, occlusion and partial damage. AprilTag's detection algorithm is optimized to detect and identify tags quickly and efficiently in real-time applications. AprilTag can be detected and identified at different scales, which means that the tag can be reliably detected no matter how far it is from the camera.

[0055] It is also important to note that each Aruco tag contains a unique ID, which is usually encoded in the black and white checkered pattern of the tag, allowing the system to identify and distinguish different tags. The Aruco tag consists of black and white squares, and this high-contrast design allows the tag to be reliably detected and identified under various lighting conditions. The four corner points of the Aruco tag are its key feature points. These corner points are used to calculate the position and posture of the tag in the image. By detecting these corner points, the translation and rotation of the tag relative to the camera can be estimated. The Aruco system is very robust and can still reliably detect and identify tags in the presence of noise, occlusion and partial damage. Aruco's detection algorithm is optimized to detect and identify tags quickly and efficiently in real-time applications. Aruco tags can be detected and identified at different scales, which means that no matter how far the tag is from the camera, it can be reliably detected.

[0056] In this possible implementation, based on the key point information, target key points in the picture taken by the target camera that are the same as the preset feature points in the calibration plate are identified. Therefore, the identified target key points are also integer data. The target key points are used as preset feature points in the picture taken by the target camera, so that subsequent calculations involving preset feature points can be calculated on the basis of integer data, thereby reducing the complexity of data calculation and reducing the cost of obtaining the picture combination calculation process.

[0057] In a possible implementation, the multiple cameras are arranged in sequence around the calibration plate; the pictures including the same preset feature points are taken as a picture combination, which can be executed according to the following steps: when the picture combination is the first picture combination, according to the preset number of pictures in the picture combination, the pictures taken by adjacent cameras are determined to be the first picture combination; when the picture combination is not the first picture combination, according to the arrangement order of the multiple cameras, the picture taken by the first camera in the current picture combination is deleted, and the picture taken by the next camera is added to the current picture combination to obtain the next picture combination.

[0058] In order to facilitate calculation, the relative positions of the camera and the calibration plate can be optimized. In a possible implementation, multiple cameras are arranged in sequence around the calibration plate. Figure 2 The camera and calibration plate are set up in a top view. Figure 2 An exemplary setting method is shown. In the figure, multiple calibration plates are surrounded by a hexagon, and the calibration plates are facing the direction of the cameras. Each camera can capture part of the information of the calibration plates. Figure 3 Shown Figure 2 A front view of a calibration plate that may be set on any surface of the device, wherein the calibration plate has rotation invariance.

[0059] In this possible implementation, each picture combination can be determined in sequence. The number of pictures in the picture combination can be predetermined, for example, it can include 2 pictures, 3 pictures or more pictures, which can be set according to actual needs, and the embodiment of the present application does not specifically limit this. When determining the first picture combination, for example, the number of pictures in the preset picture combination is 2, then two pictures taken by adjacent cameras can be arbitrarily selected from the pictures taken by multiple cameras as the first picture combination, that is, when the picture combination is the first picture combination, according to the preset number of pictures in the picture combination, the pictures taken by adjacent cameras are determined to be the first picture combination.

[0060] It should be noted that if Figure 2 As shown, camera A can capture the content of the first and second surfaces, and camera B can capture the content of the second surface. Therefore, cameras A and B can capture the same preset feature points belonging to the second surface. Therefore, the pictures taken by the adjacent cameras A and B can be used as the first picture combination. It should also be noted that the first picture combination can be pictures taken by any selected adjacent cameras, and the embodiment of the present application does not specifically limit this. For example, camera C can capture the content of the second and third surfaces, and camera D can capture the content of the third surface. Therefore, cameras C and D can capture the same preset feature points. The first picture combination can also be pictures taken by camera C and camera D. Similarly, it can also be pictures taken by other adjacent cameras.

[0061] After the first picture combination has been determined, when determining the second picture combination, it is not the first picture combination, therefore, the picture taken by the first camera in the current picture combination can be deleted according to the arrangement order of the cameras, and the picture taken by the next camera can be added to the current picture combination to obtain the next picture combination. For example, the first picture combination includes pictures taken by cameras A and B. Based on the first picture combination, the pictures taken by cameras B and camera C can be used as the second picture combination. For subsequent picture combinations, refer to the determination method of the second picture combination, and the embodiments of the present application will not be repeated here.

[0062] In this possible implementation, the picture combination taken by adjacent cameras is used as the first picture combination, which can improve the matching efficiency of the picture combination.

[0063] In a possible implementation, determining the first point data of the picture in the picture combination based on the same preset feature points can be performed according to the following steps: when the picture combination is the first picture combination, the same preset feature points are converted into three-dimensional point data using the intrinsic parameters of the camera in the picture combination, so as to obtain the first point data of each picture in the picture combination; when the picture combination is not the first picture combination, the same preset feature points are converted into three-dimensional point data using the intrinsic parameters of the camera corresponding to the picture added later in the picture combination, so as to obtain the first point data of the picture added later.

[0064] In this possible implementation, for example, it currently includes multiple picture combinations, wherein the first picture combination includes pictures taken by camera A and pictures taken by camera B, and the picture combination includes the same preset feature points. The internal parameters of camera A are used to perform dimension conversion on the same preset feature points to obtain the first point data of the picture taken by camera A. The internal parameters of camera B are used to perform dimension conversion on the same preset feature points to obtain the first point data of the picture taken by camera B. That is, when the picture combination is the first picture combination, the internal parameters of the cameras in the picture combination are used to convert the same preset feature points into three-dimensional point data to obtain the first point data of each picture in the picture combination.

[0065] Calculate the first point data of the pictures in the picture combination in sequence. After completing the calculation of the first point data of each picture in the first picture combination, for the second picture combination, include the pictures added later. Among them, the pictures added later are pictures in the previous picture combination that is different from the current picture combination. For example, the first picture combination includes pictures taken by camera A and camera B, and the second picture combination includes pictures taken by camera B and camera C, then the picture taken by camera C is the picture added later in the second picture combination. Using the internal parameters of camera C, the same preset feature points in the second picture combination are converted into three-dimensional point data to obtain the first point data of the picture taken by camera C, that is, when the picture combination is not the first picture combination, the camera internal parameters corresponding to the picture added later in the picture combination are used to convert the same preset feature points into three-dimensional point data to obtain the first point data of the picture added later.

[0066] In this possible implementation, the camera intrinsic parameters and the same feature points in each picture combination are used to calculate the three-dimensional point data of the pictures in each picture combination, that is, the first point data, to provide data support for the calculation process of the camera extrinsic parameters, so as to obtain more accurate camera extrinsic parameters.

[0067] In a possible implementation, the determining the positional relationship between the corresponding cameras of the picture combination based on the same preset feature points, the first point data and the internal parameters can be performed according to the following steps: when the picture combination is the first picture combination, taking the camera coordinate system of the specified camera in the picture combination as the reference coordinate system, determining the position of the specified camera, and determining the positional relationship of other cameras except the specified camera relative to the specified camera; when the picture combination is not the first picture combination, determining the positional relationship of the camera of the later added picture in the non-first picture combination relative to the determined external parameter camera.

[0068] In this possible implementation, when the picture combination is the first picture combination, the camera coordinate system of the designated camera in the picture combination is used as the reference coordinate system to determine the position of the designated camera, and to determine the positional relationship of other cameras relative to the designated camera except the designated camera, wherein the designated camera may be the first camera determined according to the camera arrangement order. For example, the first picture combination includes pictures taken by cameras A, B, and C, and according to the same preset feature points, first point data, and camera intrinsic parameters of these pictures, the camera coordinate system of the first camera, i.e., camera A, is used as the reference coordinate system to determine the position of camera A, the position of camera B relative to camera A, and the position of camera C relative to camera A, and obtain the positional relationship between camera B and camera C and camera A, respectively.

[0069] When the picture combination is not the first picture combination, determine the positional relationship of the camera of the later added picture in the non-first picture combination relative to the determined external reference camera. For example, for the second picture combination, including pictures taken by camera B, camera C and camera D, the later added picture is the picture taken by camera D. Therefore, determine the position of camera D relative to camera B or camera C. Since the positional relationship between camera B and camera C and camera A has been obtained, the positional relationship of camera D relative to camera A can be obtained. For subsequent picture combinations, the processing method of the second picture combination can be referred to, and no further details are given here.

[0070] In this possible implementation, the positional relationship between each camera relative to a designated camera may be determined respectively, so as to provide data support for calculating the external parameters of the camera and improve the accuracy of the external parameters.

[0071] In a possible implementation, determining the positional relationship between the cameras corresponding to the picture combination based on the same preset feature points, the first point data and the internal parameters, and calculating the external parameters of the multiple cameras based on the positional relationship can be performed according to the following steps: obtaining initial external parameters of the multiple cameras based on the same preset feature points, the first point data and the internal parameters; adjusting the same preset feature points, the first point data and the initial external parameters based on the internal parameters, and using the adjusted external parameters as the external parameters of the multiple cameras.

[0072] In this possible implementation, based on the same preset feature points, the first point data and the internal parameters, calculations are first performed to obtain initial external parameters of the multiple cameras, and the obtained initial external parameters are globally optimized. In the process of global optimization, the internal parameter values ​​remain unchanged, and the same preset feature points, the first point data and the external parameter values ​​of the multiple cameras will affect each other and change. In the process of change, a better camera external parameter is determined, that is, based on the internal parameters, the same preset feature points, the first point data and the initial external parameters are adjusted, and the adjusted external parameters are used as the external parameters of the multiple cameras.

[0073] In this possible implementation, global optimization is performed on the camera extrinsic parameters initially calculated to prevent local overfitting.

[0074] Considering that the real scale of external parameters is required in application scenarios such as three-dimensional reconstruction, robot navigation and positioning, augmented reality, measurement and detection. In applications that require real scale, the accuracy and scale consistency of external parameters are crucial because they directly affect the accuracy and reliability of the task. Therefore, in a possible implementation, after calculating the external parameters of the multiple cameras, the following steps can also be performed: obtain the second point data corresponding to the same preset feature point; wherein the second point data is located in the world three-dimensional coordinate system; calculate the conversion relationship between the camera three-dimensional coordinate system and the world three-dimensional coordinate system based on the same preset feature point, the first point data, and the second point data, and align the external parameters according to the conversion relationship.

[0075] In this possible implementation, the actual distances involved in the same preset feature points can be obtained by measuring the distances involved in the preset feature points on the calibration plate, such as the distances between the preset feature points in the calibration plate, the distances between the preset feature points and the camera, the distances between the cameras, etc. The three-dimensional points in the world coordinate system corresponding to the same preset feature points are determined based on the actual distances, i.e., the second data points. The three-dimensional points in the world three-dimensional coordinate system are used to describe the position of the camera in the global coordinate system.

[0076] Based on the same preset feature points, the first point data, and the second point data, the conversion relationship between the camera three-dimensional coordinate system and the world three-dimensional coordinate system is calculated, that is, the scale between the two coordinate systems is obtained, and then the external parameters of multiple cameras are aligned.

[0077] In this possible implementation, the real scales corresponding to the external parameters of multiple cameras can be obtained. In the three-dimensional reconstruction task, it is necessary to reconstruct the real three-dimensional model of the object or scene, which requires the external parameters of the camera to have a real scale. Only when the external parameters have a real scale can the points in the camera coordinate system be accurately converted to the world coordinate system and the real three-dimensional structure be obtained. In the robot navigation and positioning task, the robot needs to know its real position and direction in the world coordinate system for path planning and obstacle avoidance. This requires the external parameters of the camera to have a real scale so that the robot can accurately understand and interact with the environment around it. In augmented reality applications, virtual objects need to be rendered and displayed according to the position of objects in the real world to achieve integration with the real environment. This requires the external parameters of the camera to have a real scale so that virtual objects can be accurately superimposed on the real scene. In some measurement and detection applications, such as size measurement, object detection and positioning, it is necessary to know the real position and size of the object in the world coordinate system. This requires the external parameters of the camera to have a real scale so that accurate measurement and detection can be performed.

[0078] In a possible implementation, the calculation of the conversion relationship between the camera three-dimensional coordinate system and the world three-dimensional coordinate system based on the same preset feature points, the first point data, and the second point data can be performed according to the following steps: determining the feature point to be processed among the same preset feature points; calculating the first spatial distance between the first point data corresponding to the feature point to be processed and the first point data corresponding to the specified feature point, and calculating the second spatial distance between the second point data corresponding to the feature point to be processed and the second point data corresponding to the specified feature point; wherein the specified feature point is any feature point among the same preset feature points except the feature point to be processed; calculating the ratio of the second spatial distance to the first spatial distance, and determining the conversion relationship between the camera three-dimensional coordinate system and the world three-dimensional coordinate system according to the ratio.

[0079] In this possible implementation, the same preset feature points include point A, point B, point C and point D, and the feature points to be processed are determined from the same preset feature points, wherein the feature point to be processed is the point currently involved in the calculation, for example, point A, and the designated feature point is any point other than the feature point to be processed among the same preset feature points, for example, one can arbitrarily select one from points B, C, and D as the feature point to be processed. The first spatial distance between the first point data corresponding to the feature point to be processed and the first point data corresponding to the designated feature point is calculated, for example, the feature point to be processed is point A, the designated feature point is point B, and the first spatial distance and the second spatial distance between points A and B are calculated. It should be noted that the points other than the feature point to be processed among the same preset feature points can be selected as designated feature points in turn, that is, point C and point D can also be respectively selected as designated feature points. It should also be noted that after completing the calculation of the spatial distance, that is, obtaining the first spatial distance and the second spatial distance, the calculation results need to be screened, that is, it is necessary to ensure that the first spatial distance is greater than 0 and the second spatial distance is greater than 0.

[0080] The distance pairs calculated based on different designated feature points are used to calculate the distance ratio in the distance pairs. For example, when the feature point to be processed is point A, for the case where point B is the designated feature point, the first spatial distance and the second spatial distance between points A and B are calculated, and the ratio of the first spatial distance to the second spatial distance is calculated. For the case where point C is the designated feature point, a distance ratio is calculated for points A and C. For the case where point D is the designated feature point, a distance ratio can also be calculated for points A and D. The multiple ratios finally obtained are analyzed and processed to obtain the conversion relationship between the camera three-dimensional coordinate system and the world three-dimensional coordinate system, that is, the ratio of the second spatial distance to the first spatial distance is calculated, and the conversion relationship between the camera three-dimensional coordinate system and the world three-dimensional coordinate system is determined according to the ratio.

[0081] In this possible implementation, by calculating the spatial distances between the two three-dimensional points corresponding to the feature points to be processed and the two three-dimensional points corresponding to the specified feature points, the distance pairs corresponding to the feature points to be processed are obtained, and then the ratio of the distances in the distance pairs is obtained. The conversion relationship is determined based on the ratio to improve the accuracy of the conversion relationship.

[0082] In a possible implementation, determining the conversion relationship between the camera three-dimensional coordinate system and the world three-dimensional coordinate system based on the ratio includes: sorting the ratios to obtain ratio sorting results; determining multiple target ratios in the ratio sorting results according to preset ratio screening conditions, and calculating the mean and standard deviation of the multiple target ratios; when the standard deviation is less than a preset standard deviation threshold, using the mean as the conversion relationship between the camera three-dimensional coordinate system and the world three-dimensional coordinate system.

[0083] In this possible implementation, the preset ratio screening condition can be used to screen some or all of the ratios in the ratio sorting result to obtain the target ratio. The specific content of the screening condition can be set according to actual needs, and the embodiment of the present application is not specifically limited to this. For example, the ratio of the specified position can be screened in the order of arrangement to obtain the target ratio, or by setting the position range, the target ratio within the range can be obtained. The ratio screening condition can also be used to ensure the validity of the screening result, for example, the result of screening ratio greater than 0 obtains the target ratio.

[0084] In this possible implementation, by sorting and screening by comparing values, redundant and noisy data are removed, and the impact of noise on the scale is reduced. By setting a standard deviation threshold, a mean with better performance can be screened out to obtain a more stable scale.

[0085] In a possible implementation, the process of determining a plurality of target ratios in the ratio sorting result according to a preset ratio screening condition may be performed according to the following steps: performing deduplication processing on the ratios in the ratio sorting result; taking the ratio between the first position and the second position in the deduplication-processed ratio sorting result as the target ratio; wherein the first position is before the second position.

[0086] In this possible implementation, by removing duplicates from the ratios in the ratio sorting results, and by setting the first position and the second position to screen the middle ratios in the ratio sorting results as the target ratios, the influence of noise can be reduced to a certain extent and the stability of the results can be improved. It should be noted that the first position and the second position can be set according to actual needs, and the embodiment of the present application does not specifically limit this. Preferably, the first position can be in the first 20% of the sorting results, and the second position can be in the last 20% of the sorting results.

[0087] In this possible implementation, the ratios in the comparison value sorting results are deduplicated to reduce the amount of calculation of unnecessary data and improve calculation efficiency. By determining two positions in the sorting results, the ratio in the middle of the sorting results is selected as the target ratio to remove the influence of noise in the actual data processing process.

[0088] See also Figure 4 The implementation flow chart of the external parameter calibration method for multiple cameras is shown, and the implementation of the method is described below with a specific embodiment.

[0089] refer to Figure 4In the A part, the input data is a single photo and internal parameters of each camera. First, based on the image feature points (the posture of feature points such as APRILTAG and ARUCO can be fully resolved, that is, it has rotation invariance, and it is sufficient to shoot a part of the calibration plate) to obtain the 2D feature points (preset feature points in the image); feature point matching is performed based on 2D feature points. Feature points such as APRILTAG are rotation invariant and have unique IDs, which can be matched very conveniently; after the 2D feature points are matched, the initial image pair is selected, and the same preset feature points are determined based on the matching relationship. 3D feature points (first point data) are generated based on the same preset feature points, and then the feature points corresponding to the remaining images are added one by one; after the 3D feature points of all images are generated, the 2D feature points, 3D feature points, and external parameters of the images are globally nonlinearly optimized, and finally the pose (rotation matrix + translation matrix) of each photo is obtained. In addition, the 2D feature points (pixel coordinates) on each RGB photo are also obtained. FEATURE 2D_per_image And the 3D point FEATURE corresponding to the 2D feature point 3D_all (XYZ). The above results constitute the 2D-3D key point three-dimensional coordinate system, that is, the camera three-dimensional coordinate system.

[0090] See also Figure 4 In the lower part of B, the distance of the real physical scale corresponding to the same preset feature point is known, and the difference between the feature point 3D coordinate system and the 3D real scale (i.e., the second point data) 3D coordinate system is one scale. The scale can be obtained by the distance between different 2D feature points in the 2D-3D feature point 3D coordinate system and the 3D real scale 3D coordinate system (i.e., the world 3D coordinate system).

[0091] Coordinate system alignment process Figure 4 In part B, the coordinate alignment process is described below. Given the feature point 3D coordinate system (i.e., the camera 3D coordinate system) and the true scale 3D coordinate system (i.e., the world 3D coordinate system), the scene is as follows Figure 5 As shown, Figure 5 The rectangular box in the figure represents the camera, and multiple cameras are arranged in sequence to take pictures of the calibration plate. The coordinate system alignment can be completed by converting the feature point three-dimensional coordinate system to the real scale three-dimensional coordinate system. The difference between the two coordinate systems is a scalar scale SCALE FEATURE_TO_REAL .

[0092] The following describes how to obtain the scalar scale. Specifically, you can follow the steps below:

[0093] a. According to the 2D feature point FEATURE_POINT in each RGB image in the 3D coordinate system of the feature point 2D_id and its corresponding 3D feature point FEATURE_POINT 3D_id, get the 3D point sequence VEC_FEATURE_POINT in the three-dimensional coordinate system of the feature point 3D_id ;

[0094] b. Based on the 3D point sequence VEC_REAL_POINT in the real scale 3D coordinate system 3D_id :

[0095] c. The 3D point sequence in the real-scale three-dimensional coordinate system needs to be reconstructed in the real-scale three-dimensional coordinate system. Based on this constraint, the 3D point sequence in step a is filtered;

[0096] d. For the 3D point sequence in the three-dimensional coordinate system of the feature point, calculate the spatial distance VEC_DISTANCE between a certain 3D point and the remaining 3D points feature_point_ID_i1_ID_i2 ; and calculate the corresponding spatial distance VEC_DISTANCE in the real scale three-dimensional coordinate system REAL_POINT_ID_i1_ID_i2 ; Among them, ID_i1!=ID_i2.

[0097] e. The spatial distance needs to be greater than 0. Based on this constraint, the intermediate distance pairs in step d are filtered;

[0098] f. Calculate the ratio of the distance between the 3D point sequence in the true scale 3D coordinate system and the 3D point sequence in the feature point 3D coordinate system, and obtain the ratio sequence, VEC_SCALE feature_TO_real ;

[0099] in, The value range of i is [0, N1), where N1 is the size of the spatial distance sequence.

[0100] g. In order to remove the influence of noise in the actual data processing process, the ratio sequence is sorted and the smallest and largest 20% are removed. At this time, the size of the spatial sequence is reduced from N1 to N2;

[0101] h. Calculate the mean and standard deviation of the ratio sequence. The mean is the required scale SCALE FEATURE_TO_REAL , standard deviation STD VEC_SCALE Less than 0.001, it indicates that the scale S is available, where:

[0102] SCALE FEATURE_TO_REAL =∑VEC_SCALE FEATURE_TO_REAL / N2.

[0103] In the above steps, a rotation-invariant calibration plate is used, and multiple cameras can be arranged on a part of the calibration plate, and calibration can be completed by taking a single photo; based on the corner points, camera intrinsic parameters, and camera extrinsic parameters identified in each photo, the accuracy is further improved through global optimization; the image corner points are fully utilized, and the correspondence between the image corner points and the three-dimensional points is combined to obtain a usable three-dimensional point sequence, and the scale can be obtained according to the ratio between the value of the true scale of the sequence and the three-dimensional point coordinate system; combined with the effective ROI (Region of Interest) of the image, the effectiveness of obtaining scale data can be further improved, thereby improving the accuracy of obtaining scale; in order to remove the influence of noise in the actual data processing process, first sorting is performed, the largest and smallest 20% are removed, and the remaining 60% of the data are averaged and the standard deviation is calculated. Generally, a standard deviation less than 0.01 indicates that the result is available, and the standard deviation of real data testing is generally within 0.001.

[0104] Based on the embodiments of the present application, when performing external parameter calibration on multiple area array cameras, only one photo needs to be taken to quickly, accurately and stably obtain the pose scale; the operability is good and the project implementation is simple; human participation is eliminated to ensure full process automation; after actual verification, the present application can resist the influence of a certain degree of noise and has high stability.

[0105] This application also provides a three-dimensional reconstruction method, see Figure 6 A flow chart of a three-dimensional reconstruction method is shown, the method comprising:

[0106] Step 601, obtaining external parameters of multiple cameras; the external parameters are obtained according to the above-mentioned external parameter calibration method for multiple cameras.

[0107] Step 602: Process the images captured by the multiple cameras using the external parameters to obtain a three-dimensional reconstruction result.

[0108] In the embodiments of the present application, 3D reconstruction refers to reconstructing the 3D shape and structure of an object or scene from a 2D image or other data. Processing images taken by multiple cameras based on the obtained external parameters of multiple cameras to perform 3D reconstruction can improve the reconstruction accuracy of the reconstruction result, simplify the calculation process, and improve robustness and reconstruction efficiency.

[0109] The present application provides a three-dimensional reconstruction method, which includes: obtaining external parameters of multiple cameras; the external parameters are obtained according to the above-mentioned external parameter calibration method of multiple cameras. The images taken by the multiple cameras are processed using the external parameters to obtain a three-dimensional reconstruction result. The external parameter calibration method of multiple cameras can use each camera to take a photo, and perform image matching processing based on the photos taken by each camera in the multiple cameras to obtain a picture combination, thereby combining the camera internal parameters to determine the positional relationship between the cameras corresponding to the picture combination, and perform external parameter calibration of multiple cameras at the same time, which is simple to operate, efficient in the calibration process, and more accurate in results.

[0110] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides an external parameter calibration device for multiple cameras. Figure 7 The figure is a structural block diagram of an external parameter calibration device for multiple cameras according to an embodiment of the present application, and the device may include:

[0111] The acquisition module 701 is used to acquire the pictures taken by each camera of the multiple cameras and the internal parameters of each camera; wherein the pictures include at least part of the preset feature points of the calibration plate, and the calibration plate is a rotationally invariant calibration plate; the combination module 702 is used to identify the preset feature points in the pictures, and take the pictures including the same preset feature points as a picture combination; the calibration module 703 is used to determine the positional relationship between the cameras corresponding to the picture combination according to the same preset feature points and the internal parameters, and calculate the external parameters of the multiple cameras according to the positional relationship.

[0112] The embodiment of the present application can use each camera to take a photo, and perform image matching processing based on the photos taken by each camera in the multiple cameras to obtain an image combination, so as to combine the camera internal parameters to determine the positional relationship between the cameras corresponding to the image combination, and perform external parameter calibration of multiple cameras at the same time. The operation is simple, the calibration process is efficient, and the results are more accurate.

[0113] In a possible implementation, the identifying of preset feature points in the pictures and taking pictures including the same preset feature points as a picture combination includes: determining first point data of the pictures in the picture combination based on the same preset feature points; wherein the first point data is located in the three-dimensional coordinate system of the camera.

[0114] In a possible implementation, determining the positional relationship between the cameras corresponding to the picture combination based on the same preset feature points and the internal parameters includes: determining the positional relationship between the cameras corresponding to the picture combination based on the same preset feature points, first point data and the internal parameters.

[0115] In a possible implementation, after calculating the external parameters of the multiple cameras, it also includes: obtaining second point data corresponding to the same preset feature point; wherein the second point data is located in the world three-dimensional coordinate system; based on the same preset feature point, the first point data, and the second point data, calculating the conversion relationship between the camera three-dimensional coordinate system and the world three-dimensional coordinate system, and aligning the external parameters according to the conversion relationship.

[0116] In a possible implementation, the identifying of preset feature points in the image includes: obtaining key point information of the image taken by the target camera; wherein the key point information is integer data; based on the key point information, identifying target key points in the image taken by the target camera that are identical to the preset feature points in the calibration plate; and using the target key points as preset feature points in the image taken by the target camera.

[0117] In a possible implementation, the multiple cameras are arranged in sequence around the calibration plate; the pictures including the same preset feature points are taken as a picture combination, including: when the picture combination is the first picture combination, according to the preset number of pictures in the picture combination, the pictures taken by adjacent cameras are determined to be the first picture combination; when the picture combination is not the first picture combination, according to the arrangement order of the multiple cameras, the picture taken by the first camera in the current picture combination is deleted, and the picture taken by the next camera is added to the current picture combination to obtain the next picture combination.

[0118] In a possible implementation, determining the first point data of the picture in the picture combination based on the same preset feature points includes: when the picture combination is the first picture combination, using the intrinsic parameters of the camera in the picture combination to convert the same preset feature points into three-dimensional point data to obtain the first point data of each picture in the picture combination; when the picture combination is not the first picture combination, using the camera intrinsic parameters corresponding to the picture added later in the picture combination to convert the same preset feature points into three-dimensional point data to obtain the first point data of the picture added later.

[0119] In a possible implementation, the positional relationship between the corresponding cameras of the picture combination is determined based on the same preset feature points, the first point data and the internal parameters, including: when the picture combination is the first picture combination, taking the camera coordinate system of the specified camera in the picture combination as the reference coordinate system, determining the position of the specified camera, and determining the positional relationship of other cameras except the specified camera relative to the specified camera; when the picture combination is not the first picture combination, determining the positional relationship of the camera of the later added picture in the non-first picture combination relative to the determined external parameter camera.

[0120] In a possible implementation, the positional relationship between the cameras corresponding to the picture combination is determined based on the same preset feature points, the first point data and the internal parameters, and the external parameters of the multiple cameras are calculated based on the positional relationship, including: obtaining initial external parameters of the multiple cameras based on the same preset feature points, the first point data and the internal parameters; based on the internal parameters, adjusting the same preset feature points, the first point data and the initial external parameters, and using the adjusted external parameters as the external parameters of the multiple cameras.

[0121] In a possible implementation, the calculation of the conversion relationship between the camera three-dimensional coordinate system and the world three-dimensional coordinate system based on the same preset feature points, the first point data, and the second point data includes: determining a feature point to be processed among the same preset feature points; calculating a first spatial distance between the first point data corresponding to the feature point to be processed and the first point data corresponding to a designated feature point, and calculating a second spatial distance between the second point data corresponding to the feature point to be processed and the second point data corresponding to the designated feature point; wherein the designated feature point is any feature point among the same preset feature points except the feature point to be processed; calculating a ratio of the second spatial distance to the first spatial distance, and determining the conversion relationship between the camera three-dimensional coordinate system and the world three-dimensional coordinate system according to the ratio.

[0122] In a possible implementation, determining the conversion relationship between the camera three-dimensional coordinate system and the world three-dimensional coordinate system based on the ratio includes: sorting the ratios to obtain ratio sorting results; determining multiple target ratios in the ratio sorting results according to preset ratio screening conditions, and calculating the mean and standard deviation of the multiple target ratios; when the standard deviation is less than a preset standard deviation threshold, using the mean as the conversion relationship between the camera three-dimensional coordinate system and the world three-dimensional coordinate system.

[0123] In a possible implementation, the method of determining a plurality of target ratios in the ratio sorting result based on a preset ratio screening condition includes: deduplicating the ratios in the ratio sorting result; taking the ratio between the first position and the second position in the ratio sorting result after deduplication as the target ratio; wherein the first position is before the second position.

[0124] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides a three-dimensional reconstruction device. Figure 8 The figure is a structural block diagram of a 3D reconstruction device according to an embodiment of the present application, and the device may include:

[0125] The external parameter module 801 is used to obtain the external parameters of multiple cameras; the external parameters are obtained according to the above method; the reconstruction module 802 is used to use the external parameters to process the images taken by the multiple cameras to obtain a three-dimensional reconstruction result.

[0126] The external parameters of the embodiment of the present application are obtained according to the external parameter calibration method of multiple cameras mentioned above. The images taken by the multiple cameras are processed using the external parameters to obtain a three-dimensional reconstruction result. The external parameter calibration method of multiple cameras can use each camera to take a photo, and perform image matching processing based on the photos taken by each camera in the multiple cameras to obtain a picture combination, thereby combining the camera internal parameters to determine the positional relationship between the cameras corresponding to the picture combination, and perform external parameter calibration of multiple cameras at the same time, which is simple to operate, efficient in the calibration process, and more accurate in results.

[0127] The functions of each module in each device in the embodiments of the present application can be found in the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.

[0128] Fig. 9 FIG. 1 is a block diagram of an electronic device used to implement an embodiment of the present application. Fig. 9 As shown, the electronic device includes: a memory 901 and a processor 902. The memory 901 stores a computer program that can be run on the processor 902. When the processor 902 executes the computer program, the method in the above embodiment is implemented. The number of the memory 901 and the processor 902 can be one or more.

[0129] The electronic device also includes:

[0130] The communication interface 903 is used to communicate with external devices and perform data exchange transmission.

[0131] If the memory 901, the processor 902 and the communication interface 903 are implemented independently, the memory 901, the processor 902 and the communication interface 903 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0132] Optionally, in a specific implementation, if the memory 901, the processor 902 and the communication interface 903 are integrated on a chip, the memory 901, the processor 902 and the communication interface 903 can communicate with each other through an internal interface.

[0133] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present application when the program is executed by a processor.

[0134] An embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the method provided in the embodiment of the present application is implemented.

[0135] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided by the embodiment of the present application.

[0136] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.

[0137] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.

[0138] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of exemplary but not limiting description, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

[0139] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0140] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0141] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0142] Any process or method described in the flow chart or otherwise described herein can be understood as a module, fragment or portion of a code representing one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0143] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or used in combination with these instruction execution systems, devices or apparatuses.

[0144] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0145] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.

[0146] The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope recorded in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for calibrating external parameters of multiple cameras, comprising: Acquire a picture taken by each camera of a plurality of cameras and an internal parameter of each camera; wherein the picture includes at least part of preset feature points of a calibration plate, and the calibration plate is a rotationally invariant calibration plate; Identify preset feature points in the pictures, and combine pictures including the same preset feature points; Determining the positional relationship between the cameras corresponding to the picture combination according to the same preset feature points and the internal parameters; According to the positional relationship, external parameters of the multiple cameras are calculated.

2. The method according to claim 1, wherein: The step of identifying the preset feature points in the pictures and combining the pictures including the same preset feature points as pictures includes: The first point data of the picture in the picture combination is determined according to the same preset feature point; wherein the first point data is located in the camera three-dimensional coordinate system.

3. The method according to claim 2, wherein: The determining, based on the same preset feature points and the internal parameters, the positional relationship between the cameras corresponding to the picture combination includes: The positional relationship between the cameras corresponding to the picture combination is determined according to the same preset feature points, the first point data and the internal parameters.

4. The method according to claim 1, wherein: The identifying the preset feature points in the image includes: Obtain key point information of the image taken by the target camera; wherein the key point information is integer data; According to the key point information, identifying the target key points in the picture taken by the target camera that are the same as the preset feature points in the calibration plate; The target key points are used as preset feature points in the picture taken by the target camera.

5. The method according to claim 1, wherein: The plurality of cameras are arranged in sequence around the calibration plate; and the pictures including the same preset feature points are used as a picture combination, comprising: When the picture combination is the first picture combination, according to a preset number of pictures in the picture combination, determining that the pictures taken by the adjacent cameras are the first picture combination; When the picture combination is not the first picture combination, the picture taken by the first camera in the current picture combination is deleted according to the arrangement order of the multiple cameras, and the picture taken by the next camera is added to the current picture combination to obtain the next picture combination.

6. The method according to claim 2, wherein: The determining the first point data of the picture in the picture combination according to the same preset feature point includes: When the picture combination is the first picture combination, the same preset feature points are converted into three-dimensional point data using the internal parameters of the cameras in the picture combination to obtain the first point data of each picture in the picture combination; When the picture combination is not the first picture combination, the same preset feature points are converted into three-dimensional point data using the camera intrinsic parameters corresponding to the picture added later in the picture combination to obtain the first point data of the picture added later.

7. The method according to claim 3, wherein: The determining, based on the same preset feature point, the first point data and the internal parameter, the positional relationship between the cameras corresponding to the picture combination includes: When the picture combination is the first picture combination, taking the camera coordinate system of the designated camera in the picture combination as the reference coordinate system, determining the position of the designated camera, and determining the positional relationship of other cameras except the designated camera relative to the designated camera; When the picture combination is not the first picture combination, the positional relationship of the camera of the later added picture in the non-first picture combination relative to the determined extrinsic reference camera is determined.

8. The method according to claim 3, wherein: Determining the positional relationship between the cameras corresponding to the picture combination according to the same preset feature point, the first point data and the internal parameter; Calculating the external parameters of the multiple cameras according to the positional relationship includes: Obtaining initial external parameters of the multiple cameras according to the same preset feature points, the first point data and the internal parameters; Based on the internal parameters, the same preset feature points, the first point data and the initial external parameters are adjusted, and the adjusted external parameters are used as the external parameters of the multiple cameras.

9. The method according to claim 3, wherein: After calculating the external parameters of the multiple cameras, the method further includes: Acquire second point data corresponding to the same preset feature point; wherein the second point data is located in the world three-dimensional coordinate system; A conversion relationship between the camera three-dimensional coordinate system and the world three-dimensional coordinate system is calculated based on the same preset feature point, the first point data, and the second point data, and the external parameters are aligned according to the conversion relationship.

10. The method according to claim 9, wherein: The calculating the conversion relationship between the camera three-dimensional coordinate system and the world three-dimensional coordinate system based on the same preset feature point, the first point data, and the second point data includes: Determining feature points to be processed among the same preset feature points; Calculating a first spatial distance between first point data corresponding to the feature point to be processed and first point data corresponding to a designated feature point, and calculating a second spatial distance between second point data corresponding to the feature point to be processed and second point data corresponding to the designated feature point; wherein the designated feature point is any feature point other than the feature point to be processed among the same preset feature points; A ratio of the second spatial distance to the first spatial distance is calculated, and a conversion relationship between the camera three-dimensional coordinate system and the world three-dimensional coordinate system is determined according to the ratio.

11. The method according to claim 10, wherein: The step of determining the conversion relationship between the camera three-dimensional coordinate system and the world three-dimensional coordinate system according to the ratio includes: Sorting the ratios to obtain a ratio sorting result; According to a preset ratio screening condition, multiple target ratios are determined in the ratio sorting results, and the mean and standard deviation of the multiple target ratios are calculated; When the standard deviation is less than a preset standard deviation threshold, the mean is used as a conversion relationship between the camera three-dimensional coordinate system and the world three-dimensional coordinate system.

12. The method according to claim 11, wherein: Determining a plurality of target ratios in the ratio sorting results according to a preset ratio screening condition includes: Deduplication processing is performed on the ratios in the ratio sorting result; The ratio between the first position and the second position in the ratio sorting result after deduplication processing is used as the target ratio; wherein the first position is located before the second position.

13. A three-dimensional reconstruction method, comprising: Get external parameters of multiple cameras; The external reference is obtained according to the method described in any one of claims 1 to 12; The images captured by the multiple cameras are processed using the external parameters to obtain a three-dimensional reconstruction result.

14. An electronic device comprising a memory, a processor and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 13 when executing the computer program.

15. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

Images