Method, device and electronic device for realizing camera calibration

Abstract

The present application discloses a method, device and computer-readable storage medium for realizing camera calibration. The specific implementation scheme is: based on the first reference pose of the rigid body relative to the motion capture camera and the first calculated pose of the calibration plate relative to the camera to be calibrated, determine the second calculated pose of the calibration plate relative to the motion capture camera and the third calculated pose of the camera to be calibrated relative to the rigid body, wherein the camera to be calibrated is installed on the rigid body, and the rigid body can drive the camera to be calibrated to move; based on the first calculated pose, the second calculated pose, and the third calculated pose, determine the fourth calculated pose of the rigid body relative to the motion capture camera; optimize the first calculated pose using the first error between the fourth calculated pose and the first reference pose to determine the optimized pose of the calibration plate relative to the camera to be calibrated; based on the optimized pose, calibrate the camera to be calibrated.

Description

Technical Field

[0001] The present disclosure relates to the field of machine vision technology, and in particular to a method, device and electronic device for implementing camera calibration. Background Art

[0002] A head-mounted display device is a device that can be worn on the user's head and has a display function, such as augmented reality glasses, virtual reality glasses, mixed reality glasses, etc. Various types of head-mounted display devices usually have image acquisition devices (such as cameras) and various sensors. Based on the data collected by the above devices, positioning algorithms are used to provide a basis for image display to improve the display effect. It is usually necessary to calibrate the camera to ensure the accuracy of the positioning algorithm. Summary of the invention

[0003] Embodiments of the present disclosure provide a method, an apparatus, and an electronic device for implementing camera calibration.

[0004] According to one aspect of the present disclosure, a method for implementing camera calibration is provided, comprising:

[0005] Based on a first reference pose of the rigid body relative to the motion capture camera and a first calculated pose of the calibration plate relative to the camera to be calibrated, a second calculated pose of the calibration plate relative to the motion capture camera and a third calculated pose of the camera to be calibrated relative to the rigid body are determined, wherein the camera to be calibrated is mounted on the rigid body, and the rigid body can drive the camera to be calibrated to move;

[0006] Determining a fourth calculated pose of the rigid body relative to the motion capture camera based on the first calculated pose, the second calculated pose, and the third calculated pose;

[0007] Optimizing the first calculated pose using a first error between the fourth calculated pose and the first reference pose to determine an optimized pose of the calibration plate relative to the camera to be calibrated;

[0008] Based on the optimized pose, the camera to be calibrated is calibrated.

[0009] According to another aspect of the present disclosure, a device for implementing camera calibration is provided, comprising:

[0010] A first determination module is used to determine a second calculated pose of the calibration plate relative to the motion capture camera and a third calculated pose of the camera to be calibrated relative to the rigid body based on a first reference pose of the rigid body relative to the motion capture camera and a first calculated pose of the calibration plate relative to the camera to be calibrated, wherein the camera to be calibrated is installed on the rigid body, and the rigid body can drive the camera to be calibrated to move;

[0011] a second determination module, configured to determine a fourth calculated pose of the rigid body relative to the motion capture camera based on the first calculated pose, the second calculated pose, and the third calculated pose;

[0012] A third determination module is used to optimize the first calculated pose by using a first error between the fourth calculated pose and the first reference pose, so as to determine an optimized pose of the calibration plate relative to the camera to be calibrated;

[0013] The calibration module is used to calibrate the camera to be calibrated based on the optimized pose.

[0014] According to another aspect of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the above-mentioned method for realizing camera calibration.

[0015] According to another aspect of the present disclosure, there is provided an electronic device, including:

[0016] processor;

[0017] a memory for storing processor-executable instructions;

[0018] The processor is used to read the executable instructions from the memory and execute the instructions to implement the above method for realizing camera calibration.

[0019] According to another aspect of the present disclosure, a computer program product is provided, including computer program instructions, and when the computer program instructions are executed by a processor, the method for realizing camera calibration is implemented.

[0020] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0022] Figure 1 It is a flowchart of a method for implementing camera calibration provided by an exemplary embodiment of the present disclosure.

[0023] Figure 2 It is a flowchart of a method for implementing camera calibration provided by another exemplary embodiment of the present disclosure.

[0024] Figure 3It is a flowchart of a method for implementing camera calibration provided by yet another exemplary embodiment of the present disclosure.

[0025] Figure 4 It is a flowchart of a method for implementing camera calibration provided by yet another exemplary embodiment of the present disclosure.

[0026] Figure 5 It is a structural schematic diagram of a device for implementing camera calibration provided by an exemplary embodiment of the present disclosure.

[0027] Figure 6 It is a structural schematic diagram of an apparatus for implementing camera calibration provided by another exemplary embodiment of the present disclosure.

[0028] Figure 7 It is a structural diagram of a device for implementing camera calibration provided by yet another exemplary embodiment of the present disclosure.

[0029] Figure 8 It is a structural schematic diagram of an apparatus for implementing camera calibration provided by yet another exemplary embodiment of the present disclosure.

[0030] Fig. 9 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] Below, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0032] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0033] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0034] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0035] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0036] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0037] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0038] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0039] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0040] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0041] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0042] The disclosed embodiments can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0043] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0044] Exemplary Methods

[0045] Figure 1 It is a flowchart of a method for implementing camera calibration provided by an exemplary embodiment of the present disclosure. Figure 1 The method shown may include step 120, step 140, step 160 and step 180, each of which is described below.

[0046] Step 120, based on the first reference pose of the rigid body relative to the motion capture camera and the first calculated pose of the calibration plate relative to the camera to be calibrated, determine the second calculated pose of the calibration plate relative to the motion capture camera and the third calculated pose of the camera to be calibrated relative to the rigid body, wherein the camera to be calibrated is installed on the rigid body, and the rigid body can drive the camera to be calibrated to move.

[0047] Optionally, the rigid body can be fixed to the end of the robot arm, for example, the rigid body can be a flange fixed to the end of the robot arm. The robot arm can be supported by a robot arm base, the robot arm base can be fixed differently, the robot arm can move relative to the robot arm base, and the end flange of the robot arm can move with the movement of the entire robot arm, thereby realizing the movement of the end flange of the robot arm relative to the robot arm base. Of course, the rigid body is not limited to the end flange of the robot arm, and the rigid body can also be fixed to other parts of the robot arm except the end.

[0048] Optionally, the camera to be calibrated can be installed on the rigid body by screwing, clamping or other methods known to those skilled in the art. In this way, when the robotic arm as a whole moves, the rigid body will move with the movement of the robotic arm as a whole, and the camera to be calibrated will move due to the drive of the rigid body.

[0049] It should be noted that a motion capture camera generally includes multiple cameras; the calibration plate can also be called a mark plate, and the calibration plate can be placed under the robotic arm.

[0050] It should be noted that any posture involved in the embodiments of the present disclosure includes position and posture (i.e., posture = position + posture). In the embodiments of the present disclosure, a motion capture camera coordinate system, a calibration board coordinate system, a rigid body coordinate system, and a camera coordinate system to be calibrated can be established. Optionally, the motion capture camera coordinate system can be coordinate system 1, the calibration board coordinate system can be coordinate system 2, the rigid body coordinate system can be coordinate system 3, and the camera coordinate system to be calibrated can be coordinate system 4. The posture of the rigid body relative to the motion capture camera can be expressed as 1T3 or 1 T3, the position of the calibration plate relative to the camera to be calibrated can be expressed as 4T2 or 4 T2, the position of the calibration plate relative to the motion capture camera can be expressed as 1T2 or 1 T2, the position of the camera to be calibrated relative to the rigid body can be expressed as 3T4 or 3 T4.

[0051] The following introduces the calibration method of the motion capture camera and the establishment method of the rigid body coordinate system.

[0052] It is understandable that the relative position and posture relationship between multiple cameras in the motion capture camera needs to be calibrated. For example, a calibration rod can be used to calibrate the relative position and posture between multiple cameras in the motion capture camera. Optionally, there can be several collinear targets on the calibration rod, wherein the distances between two targets are not equal. By continuously waving the calibration rod, the calibration rod can move in an area that can be captured by multiple cameras in the motion capture camera. Multiple cameras in the motion capture camera all shoot the calibration rod. There are luminous points in the images captured by multiple cameras in the motion capture camera. The luminous points can represent the position of the target in the image. Since the distances between several targets are not equal, different luminous points correspond to different targets. In addition, the coordinate system of one of the multiple cameras in the motion capture camera can be defined as a world coordinate system. This camera can be called camera No. 0. According to the luminous points in the images captured by multiple cameras in the motion capture camera, the coordinates of several targets in camera No. 0 (world coordinate system) and the relative position and posture between multiple cameras in the motion capture camera can be obtained by triangulation and optimization. The exit condition of the optimization can be: the reprojection error (the difference between the projection point of the light-emitting point in the camera calculated based on the relative position and posture calculated in the previous step and the projection point actually photographed by the camera) is less than the threshold. In this way, the position and posture of the cameras other than camera 0 in the motion capture camera relative to camera 0 (world coordinate system) are obtained.

[0053] For example, there may be three balls on the calibration rod, the three balls are located on the same straight line, and the distances between the three balls are unequal to form a long side and a short side, and the lengths of the long side and the short side may be known. The long side is formed based on the ball located in the middle and the ball located at one edge of the three balls, and the short side is formed based on the ball located in the middle and the ball located at the other edge of the three balls.

[0054] Next, the world coordinate system global can be established based on the world coordinate system in the above text. For example, the world coordinate system can be transferred from camera 0 to a certain position using a triangular calibration ruler with a very flat surface. Optionally, the triangular calibration ruler does not move and is placed at this position. A small ball can be distributed on each of the three vertices of the triangular calibration ruler. In this way, there are a total of three small balls on the triangular calibration ruler. A right triangle can be formed based on these three small balls. The distances between the right angle point and the two non-right angle points of the right triangle may not be equal, and the distances between the right angle point and the two non-right angle points may be known. According to the images of the triangular calibration ruler taken by multiple cameras in the motion capture camera, and the relative position and posture relationship between the multiple cameras in the motion capture camera obtained in the above text, the position and posture of the triangular calibration ruler in camera 0 (world coordinate system) can be obtained by triangulation. By taking the right angle point of the triangular ruler as the origin, the long side of the triangular ruler as the x-axis, the short side of the triangular ruler as the y-axis, and multiplying the x-axis by the y-axis to get the z-axis, a new world coordinate system global can be formed. In this way, the position and posture of multiple cameras in the motion capture camera in the world coordinate system global can be obtained.

[0055] After that, a rigid body coordinate system can be established. Optionally, the rigid body can be fixed to the end of the robotic arm, and the robotic arm remains stationary, and the rigid body also remains stationary. There can be several non-coplanar targets on the rigid body, for example, there are 4 small balls on the rigid body, and the 4 small balls may not be coplanar. Since the position and posture of multiple cameras in the motion capture camera in the global world coordinate system have been obtained in the previous paragraph, the coordinates of several targets in the global world coordinate system can be obtained by using triangulation. According to the coordinates of several targets in the global world coordinate system, a coordinate system can be established on the rigid body, that is, the origin and three coordinate axes are determined, thereby realizing the establishment of a rigid body coordinate system.

[0056] In step 120, the first reference pose and the first calculated pose can be obtained first. The first reference pose can be the pose of the rigid body relative to the motion capture camera (assuming it is pose X). The first calculated pose can be obtained by calculation. The first calculated pose can be the pose of the calibration plate relative to the camera to be calibrated when the rigid body is in pose X relative to the motion capture camera. Next, the hand-eye calibration algorithm can be used to calculate based on the first reference pose and the first calculated pose, so as to determine the second calculated pose and the third calculated pose.

[0057] Step 140: Determine a fourth calculated pose of the rigid body relative to the motion capture camera based on the first calculated pose, the second calculated pose, and the third calculated pose.

[0058] In step 140, a hand-eye calibration algorithm may be used to perform calculations based on the first calculated pose, the second calculated pose, and the third calculated pose, so as to determine a fourth calculated pose.

[0059] Step 160: Optimize the first calculated pose using a first error between the fourth calculated pose and the first reference pose to determine an optimized pose of the calibration plate relative to the camera to be calibrated.

[0060] It should be noted that the first reference pose can be regarded as the observed value of the pose of the rigid body relative to the motion capture camera, and the fourth calculated pose can be regarded as the estimated value of the pose of the rigid body relative to the motion capture camera. The estimated value and the observed value of the pose of the rigid body relative to the motion capture camera can both be represented in the form of matrices, so that the first error can be obtained by comparing the two matrices.

[0061] In step 160, the first calculated pose can be used as a variable to be optimized, and a specified optimization algorithm can be used to perform optimization processing with minimizing the first error as the optimization goal to achieve the optimization of the first calculated pose. As described above, it can be considered that the first reference pose is relatively accurate, and the fourth calculated pose is optimized using this value. During the optimization process, the value used to calculate the fourth calculated pose can be optimized together, so that the optimized pose of the calibration plate relative to the camera to be calibrated can be determined. Optionally, the specified optimization algorithm can be a nonlinear optimization algorithm.

[0062] It should be pointed out that when optimizing by using a specified optimization algorithm and taking minimizing the first error as the optimization goal, in addition to optimizing the first calculated pose, the second calculated pose and the third calculated pose can also be optimized.

[0063] Step 180: calibrate the camera to be calibrated based on the optimized posture.

[0064] In step 180 , based on the optimized posture, any feasible method may be used to calibrate the intrinsic parameters of the camera to be calibrated.

[0065] In the embodiments of the present disclosure, the second calculated pose of the calibration plate relative to the motion capture camera and the third calculated pose of the camera to be calibrated relative to the rigid body can be determined based on the first reference pose of the rigid body relative to the motion capture camera and the first calculated pose of the calibration plate relative to the camera to be calibrated, and the fourth calculated pose of the rigid body relative to the motion capture camera can be determined based on the first calculated pose, the second calculated pose, and the third calculated pose, so that the estimated value and observed value of the pose of the rigid body relative to the motion capture camera are obtained. Afterwards, by forming an error amount between the estimated value and the observed value of the pose of the rigid body relative to the motion capture camera, and optimizing the pose with reference to the error amount, the optimized pose of the calibration plate relative to the camera to be calibrated can be determined, and the camera to be calibrated can be calibrated using the determined optimized pose, which can effectively improve the accuracy and reliability of the calibration result of the camera to be calibrated, thereby providing an accurate basis for the positioning algorithm that needs to use the image data collected by the camera.

[0066] exist Figure 1 Based on the embodiment shown, Figure 2 As shown, before step 120 , the method further includes: step 102 , step 104 and step 106 .

[0067] Step 102 , when the rigid body drives the camera to be calibrated to move, multiple cameras in the motion capture camera are controlled to respectively capture a first image sequence of the rigid body, and the camera to be calibrated is controlled to capture a second image sequence of the calibration plate.

[0068] Optionally, a motion trajectory can be set for the rigid body in advance. For example, a number of (e.g., 8, 10, 15, 20, etc.) trajectory points can be predetermined, each of which can be characterized by a position of the rigid body relative to the motion capture camera, and then the motion trajectory is determined by a number of trajectory points. Optionally, a number of trajectory points can all be located on the motion trajectory, so that in step 102, the rigid body can be controlled to move along a pre-set motion trajectory, so that the rigid body drives the camera to be calibrated to move along the corresponding trajectory. Of course, it is also possible not to set a motion trajectory for the rigid body in advance, but to move the rigid body along a random motion trajectory, so that the rigid body drives the camera to be calibrated to move along the corresponding trajectory. It should be pointed out that, regardless of whether the rigid body moves along a pre-set motion trajectory or a random motion trajectory, it is necessary to ensure that the rigid body is always within the area that the motion capture camera can capture, and the calibration plate is always within the area that the camera to be calibrated can capture.

[0069] Step 104: acquiring a first pose sequence based on first image sequences corresponding to each of the plurality of cameras in the motion capture camera, and determining a first reference pose based on the first pose sequence.

[0070] In step 104, based on the first image sequences corresponding to the multiple cameras in the motion capture camera, any feasible method can be used to determine the first pose sequence. Assume that the motion capture camera has three cameras, namely, camera 1, camera 2, and camera 3. The first image sequence corresponding to camera 1 is I 11 ,I 12 ,I 13 ,I 14 ,I 15 , the first image sequence corresponding to camera 2 is I 21 ,I 22 ,I 23 ,I 24 ,I 25 , the first image sequence corresponding to camera 3 is I 31 ,I 32 ,I 33 ,I 34 ,I 35 . Then we can base on I 11 ,I 21 ,I 31 Determine the position S1 of the rigid body relative to the motion capture camera, based on I 12 ,I 22 ,I 32 Determine another pose S2 of the rigid body relative to the motion capture camera, based on I 13 ,I 23 ,I 33 Determine the rigid body relative to the motion capture camera and then position S3, based on I 14 ,I 24 ,I 34 Determine another position S4 of the rigid body relative to the motion capture camera, based on I 15 ,I 25 ,I 35 Determine another posture S5 of the rigid body relative to the motion capture camera, and the sequentially arranged S1, S2, S3, S4, and S5 can form a first posture sequence. It should be noted that the first posture sequence can also include only some of the postures in S1, S2, S3, S4, and S5, or more postures can be obtained by performing posture interpolation processing on the basis of S1, S2, S3, S4, and S5, and S1, S2, S3, S4, S5 and more postures are sequentially arranged to form the first posture sequence.

[0071] Next, the first reference pose can be determined based on the first pose sequence. The first reference pose is determined based on the first pose sequence, which can be understood as the first reference pose only including the pose of the rigid body in the pose sequence relative to the motion capture camera when the rigid body moves to the positions of several trajectory points, that is, only including the pose in the first pose sequence. Or it can be understood that the first reference pose includes the pose of the rigid body in the pose sequence relative to the motion capture camera when the rigid body moves to the positions of several trajectory points, and the pose of the rigid body in the pose sequence relative to the motion capture camera when the rigid body moves to the positions between the positions of each trajectory point, that is, including the poses except the first pose sequence.

[0072] Step 106: Based on the second image sequence, a second posture sequence is acquired, and the first calculated posture is determined based on the second posture sequence.

[0073] In step 106, a PNP (perspective-n-point) algorithm can be used to determine the pose corresponding to each image in the second image sequence. Each determined pose is the pose of the calibration plate relative to the camera to be calibrated. These determined poses can be arranged in sequence to form a second pose sequence, and the first calculated pose can be determined based on the second pose sequence.

[0074] The first calculated pose is determined based on the second pose sequence, which can be understood as the first calculated pose only including the poses in the second pose sequence. Alternatively, it can be understood as the first calculated pose including the poses in the second pose sequence and the poses in the pose sequence of the calibration plate relative to the camera to be calibrated between adjacent image acquisition operations of the camera to be calibrated, that is, including the poses except the second pose sequence.

[0075] It should be noted that when optimizing the posture of a reference rigid body relative to a motion capture camera based on the error between the estimated value and the observed value, a large amount of sample data is generally required, and each set of sample data needs to include a first reference position and a first calculated posture. In view of this, in the embodiments of the present disclosure, the rigid body can be imaged at a fixed frequency by multiple cameras in the motion capture camera to acquire a first posture sequence, and the calibration plate can be imaged at another fixed frequency by the camera to be calibrated to acquire a second posture sequence. Based on the first posture sequence and the second posture sequence, a large amount of sample data can be acquired so that a large amount of sample data can be used for posture optimization, which is conducive to ensuring the accuracy and reliability of the calibration results of the camera to be calibrated. In addition, since the first reference pose is determined based on the first pose sequence, the first calculated pose is determined based on the second pose sequence, and the first pose sequence is obtained by acquiring images based on multiple cameras in the motion capture camera when the rigid body drives the camera to be calibrated to move, and the second pose sequence is obtained by acquiring images based on the camera to be calibrated when the rigid body drives the camera to be calibrated to move, the captured images of multiple cameras in the motion capture camera can be used to constrain the trajectory of the camera to be calibrated, thereby making the trajectory of the camera to be calibrated closer to the actual situation, which is beneficial to further ensure the accuracy and reliability of the calibration results of the camera to be calibrated.

[0076] In an optional example, the first image sequence is obtained by capturing images of a rigid body at a first frequency, and the second image sequence is obtained by capturing images of a calibration plate at a second frequency, the second frequency is different from the first frequency, and the first pose sequence corresponds to the first frequency, and the second pose sequence corresponds to the second frequency.

[0077] Here, the second frequency may be higher or lower than the first frequency. For example, the first frequency may be 100 Hz and the second frequency may be 70 Hz. Alternatively, the first frequency may be 60 Hz and the second frequency may be 80 Hz.

[0078] exist Figure 2 Based on the embodiment shown, Figure 3 As shown, the method further includes step 108 and step 110.

[0079] Step 108: Determine a first time difference between the second posture sequence and the first posture sequence.

[0080] It should be noted that the rigid body can be a flange fixed to the end of the robotic arm, and the rigid body moves with the overall movement of the robotic arm. In this way, the first pose sequence may correspond to the time system of the robotic arm's movement, and the second pose sequence may correspond to the time system of the camera to be calibrated. These two time systems may not be synchronized. For example, there may be a difference between the initial time of the robotic arm's movement and the initial time of the camera to be calibrated. Therefore, in step 108, the time difference between the two time systems can be determined (for example, the difference between the two initial times can be determined), and the time difference can be used as the first time difference between the second pose sequence and the first pose sequence.

[0081] Step 110, based on the first time difference, perform a first time synchronization on the first pose sequence and the second pose sequence, determine the first reference pose based on the first pose sequence after the first time synchronization, determine the first calculated pose based on the second pose sequence after the first time synchronization, and align the first calculated pose with the first reference pose in time.

[0082] In step 110, based on the first time difference, the time system of the robot arm's movement can be aligned in time with the time system of the camera taking pictures to be calibrated. For example, it can be determined that 0 seconds in the time system of the robot arm's movement corresponds to the second in the time system of the camera taking pictures to be calibrated, and 2 seconds in the time system of the robot arm's movement corresponds to the second in the time system of the camera taking pictures to be calibrated, and so on.

[0083] In an alternative example, the first pose sequence is as follows:

[0084] X1, X2, X3, X4, X5, X6

[0085] Among them, for the time system of the robot arm movement, the time corresponding to X1 is 0:00 (ie, zero second), the time corresponding to X2 is 1:00 (ie, the first second), the time corresponding to X3 is 2:00, the time corresponding to X4 is 3:00, the time corresponding to X5 is 4:00, and the time corresponding to X6 is 5:00.

[0086] The second pose sequence is as follows:

[0087] Y1, Y2, Y3, Y4, Y5, Y6

[0088] Among them, for the time system when the camera to be calibrated takes pictures, the time corresponding to Y1 is 00:00, the time corresponding to Y2 is 1:00, the time corresponding to Y3 is 2:00, the time corresponding to Y4 is 3:00, the time corresponding to Y5 is 4:00, and the time corresponding to Y6 is 5:00.

[0089] Assuming that the first time difference is 1 second, when the first pose sequence and the second pose sequence are synchronized for the first time, it can be determined that 0:00 in the time system of the robot arm movement corresponds to 1:00 in the time system of the camera to be calibrated. It can be further determined that X1 in the first pose sequence is aligned with Y2 in the second pose sequence in time, X2 in the first pose sequence is aligned with Y3 in the second pose sequence in time, X3 in the first pose sequence is aligned with Y4 in the second pose sequence in time, X4 in the first pose sequence is aligned with Y5 in the second pose sequence in time, and X5 in the first pose sequence is aligned with Y6 in the second pose sequence in time. In this way, when X1 is used as the first reference pose, Y2 can be used as the first calculated pose; when X2 is used as the first reference pose, Y3 can be used as the first calculated pose; when X3 is used as the first reference pose, Y4 can be used as the first calculated pose; and the same is true for the subsequent steps, which will not be repeated here.

[0090] It can be seen that the time difference between different time systems is taken into account in the embodiments of the present disclosure. By performing the first time synchronization of the first pose sequence and the second pose sequence based on the time difference, the time corresponding to each pose in the first pose sequence can be converted into the time system of the camera to be calibrated for taking pictures, or the time corresponding to each pose in the second pose sequence can be converted into the time system of the movement of the robotic arm, so as to obtain the first reference pose and the first calculated pose that are aligned in time. In this way, when the first reference pose and the first calculated pose are used for calibrating the camera to be calibrated, the accuracy and reliability of the calibration results can be effectively guaranteed.

[0091] In an optional example, determining a first time difference between the second pose sequence and the first pose sequence includes:

[0092] Extracting a first posture sequence from the first posture sequence;

[0093] extracting a second posture sequence from the second posture sequence;

[0094] The posture sequence with a higher corresponding frequency in the first posture sequence and the second posture sequence is used as a reference posture sequence, and another posture sequence in the first posture sequence and the second posture sequence is interpolated to obtain an interpolated posture sequence, wherein the frequency corresponding to the interpolated posture sequence is the same as that of the reference posture sequence;

[0095] Based on the reference pose sequence and the interpolated pose sequence, a first time difference between the second pose sequence and the first pose sequence is determined.

[0096] It should be noted that each posture in the first posture sequence may include a position and a posture, so the posture may be extracted from each posture in the first posture sequence, and these postures may be arranged in sequence to form the first posture sequence. In a similar manner, the second posture sequence may be extracted from the second posture sequence, so that the first posture sequence and the first posture sequence may correspond to the same frequency (i.e., both correspond to the first frequency), and the second posture sequence and the second posture sequence may correspond to the same frequency (i.e., both correspond to the second frequency).

[0097] Next, the first frequency corresponding to the first posture sequence can be compared with the second frequency corresponding to the second posture sequence. If the first frequency is higher than the second frequency, the first posture sequence can be used as a reference posture sequence, and if the first frequency is lower than the second frequency, the second posture sequence can be used as a reference posture sequence. For another posture sequence in the first posture sequence and the second posture sequence, an interpolation posture sequence having the same corresponding frequency as the reference posture sequence can be obtained by interpolating the second posture sequence. For example, if the first frequency is 100 Hz and the second frequency is 70 Hz, an interpolation posture sequence having a corresponding frequency of 100 Hz can be obtained by interpolating the second posture sequence.

[0098] Optionally, the interpolation process in the embodiment of the present disclosure may be spherical linear interpolation (Slerp), and the formula involved in Slerp may be in the following form:

[0099] Slerp(q0,q1,t)=q0(q0 -1 q1) t

[0100] =q1(q1 -1 q0) 1-t

[0101] =(q0q1 -1 ) 1-t q1

[0102] =(q1q0 -1 ) t q0

[0103] Among them, q0 and q1 represent the starting point and end point of the arc respectively, and t represents a parameter and is between 0 and 1.

[0104] After obtaining an interpolated posture sequence having the same corresponding frequency as the reference posture sequence, a first time difference between the second posture sequence and the first posture sequence can be determined based on the reference posture sequence and the interpolated posture sequence. In an optional embodiment, determining the first time difference between the second posture sequence and the first posture sequence based on the reference posture sequence and the interpolated posture sequence includes:

[0105] Performing differentiation processing on the reference attitude sequence to obtain a first angular velocity sequence;

[0106] Performing differentiation processing on the interpolated attitude sequence to obtain a second angular velocity sequence;

[0107] Determine a second time difference between the first angular velocity sequence and the second angular velocity sequence using a cross-correlation algorithm;

[0108] Based on the second time difference, a first time difference between the second pose sequence and the first pose sequence is determined.

[0109] Here, by performing differential processing on the reference posture sequence, the angular velocity corresponding to each posture in the reference posture sequence can be obtained, and these angular velocities can be arranged in sequence to form a first angular velocity sequence. Moreover, the first angular velocity sequence and the reference posture sequence can correspond to the same frequency. In a similar manner, a second angular velocity sequence can be obtained, and the second angular velocity sequence and the interpolation posture sequence can correspond to the same frequency. Since the reference posture sequence and the interpolation posture sequence correspond to the same frequency, the first angular velocity sequence and the second angular velocity sequence will also correspond to the same frequency.

[0110] It should be pointed out that since the rigid body is a flange fixed to the end of the robotic arm and the camera to be calibrated is installed on the rigid body, it can be considered that the rigid body, the camera to be calibrated and the end of the robotic arm form a rigid body system. Since they belong to the same rigid system, the coordinate system established based on the rigid body (i.e. the rigid body coordinate system mentioned above) and the coordinate system established based on the camera to be calibrated (i.e. the camera coordinate system to be calibrated mentioned above) have the same angular velocity modulus at the same time.

[0111] Next, the cross-correlation algorithm can be used to determine the second time difference between the first angular velocity sequence and the second angular velocity sequence. For example, the difference corresponding to the maximum value of the cross-correlation response can be determined, and the difference can be used as the second time difference. Optionally, in the process of determining the time difference using the cross-correlation algorithm, Fourier transform can be used for accelerated calculation. Afterwards, the first time difference between the second posture sequence and the first posture sequence can be determined based on the second time difference.

[0112] In this implementation, by differential processing of the posture sequence combined with the use of a cross-correlation algorithm, the second time difference can be determined efficiently and quickly, so as to thereby determine the first time difference.

[0113] In an optional implementation, determining a first time difference between the second posture sequence and the first posture sequence based on the second time difference includes:

[0114] Based on the second time difference, performing a second time synchronization on the first pose sequence and the second pose sequence;

[0115] Determine a second reference pose of the rigid body relative to the motion capture camera based on the first pose sequence after second time synchronization;

[0116] Based on the second pose sequence after the second time synchronization, determining a fifth calculated pose of the calibration plate relative to the camera to be calibrated, the fifth calculated pose being aligned with the second reference pose in time;

[0117] Determining a sixth calculated pose of the rigid body relative to the motion capture camera based on the second reference pose and the fifth calculated pose;

[0118] optimizing the second time difference using a second error between the sixth calculated posture and the second reference posture to determine an optimized time difference between the first angular velocity sequence and the second angular velocity sequence;

[0119] The optimized time difference is determined as a first time difference between the second pose sequence and the first pose sequence.

[0120] It should be noted that, based on the second time difference, the first pose sequence and the second pose sequence are synchronized in the second time, and the method of determining the second reference pose and the fifth calculated pose aligned in time can be referred to the relevant description of step 110 above, which will not be repeated here.

[0121] It should be noted that the method for determining the sixth calculated posture based on the second reference posture and the fifth calculated posture can be determined by referring to the above description of the method for determining the fourth calculated posture of the rigid body relative to the motion capture camera based on the first reference posture and the first calculated posture (specifically, refer to the description of step 120 and step 140), which will not be repeated here.

[0122] It should be noted that the second reference pose can be regarded as the observed value of the pose of the rigid body relative to the motion capture camera, and the sixth calculated pose can be regarded as the estimated value of the pose of the rigid body relative to the motion capture camera. The estimated value and the observed value of the pose of the rigid body relative to the motion capture camera can both be represented in the form of matrices, so that the second error can be obtained by comparing the two matrices.

[0123] Since the determination of the second reference posture utilizes the first posture sequence after the second time synchronization, the determination of the fifth calculated posture utilizes the second posture sequence after the second time synchronization, and the implementation of the second time synchronization is based on the second time difference, it can be considered that the determination of the second reference posture and the sixth calculated posture both need to rely on the second time difference, and the second error between the sixth calculated posture and the second reference posture is naturally also related to the second time difference. In view of this, the second time difference can be used as a variable to be optimized, and a specified optimization algorithm can be used to minimize the second error as the optimization goal for optimization processing to achieve the optimization of the second time difference, thereby determining the optimized time difference between the first angular velocity sequence and the second angular velocity sequence, and the determined optimized time difference can be used as the first time difference. Optionally, the specified optimization algorithm can be a nonlinear optimization algorithm.

[0124] In this implementation, an error can be formed between the estimated value and the observed value of the position and posture of the rigid body relative to the motion capture camera, and the time difference is optimized with reference to the error to determine the optimized time difference between the first angular velocity sequence and the second angular velocity sequence, so as to determine the first time difference accordingly. This can effectively ensure the accuracy and reliability of the determined first time difference, so that when the first time synchronization is performed based on the first time difference, the accuracy and reliability of the first time synchronization result can be guaranteed, thereby further ensuring the time alignment of the first reference posture determined based on the first time synchronization result and the first calculated posture, thereby ensuring the accuracy and reliability of the calibration result of the camera to be calibrated.

[0125] Of course, the manner of determining the first time difference based on the second time difference is not limited thereto. For example, the second time difference may be directly determined as the first time difference, which is also feasible.

[0126] In the embodiments of the present disclosure, by extracting posture sequences for the first posture sequence and the second posture sequence respectively, and by comparing the frequencies corresponding to the two posture sequences, the reference posture sequence can be determined efficiently and quickly, and the interpolated posture sequence can be obtained by interpolation processing with the reference posture sequence as a reference, so as to refer to the reference posture sequence and the interpolated posture sequence with the same corresponding frequency, so as to efficiently and quickly realize the determination of the first time difference.

[0127] exist Figure 1 Based on the embodiment shown, Figure 4 As shown, step 160 includes step 1601, step 1603, step 1605 and step 1607.

[0128] Step 1601, extracting a first position and a first posture from a fourth calculated posture.

[0129] Since each posture involved in the embodiments of the present disclosure includes a position and a posture, in step 1601, the position and posture can be extracted for the fourth calculated posture, and the extracted position can be used as the first position, and the extracted posture can be used as the first posture.

[0130] Step 1603, extracting a second position and a second posture from the first reference posture.

[0131] Similar to step 1601, in step 1603, the position and posture may be extracted for the first reference posture, and the extracted position may be used as the second position, and the extracted posture may be used as the second posture.

[0132] Step 1605: Optimize the third position in the first calculated pose using the error between the first position and the second position to determine the optimal position of the calibration plate relative to the camera to be calibrated.

[0133] Step 1607: Optimize the third posture in the first calculated posture using the error between the first posture and the second posture to determine the optimized posture of the calibration plate relative to the camera to be calibrated.

[0134] It should be noted that the error between the first position and the second position, and the error between the first posture and the second posture can together constitute the first error mentioned above. Based on the first error, the third position and the third posture in the first calculated posture can be optimized respectively, so as to determine the optimized position and optimized posture of the calibration plate relative to the camera to be calibrated, and the determined optimized position and optimized posture can together constitute the optimized posture mentioned above.

[0135] Optionally, assuming that Δ1P3 represents the error between the first position and the second position, and Δ1C3 represents the error between the first posture and the second posture, then:

[0136] Δ1P3=1C2(4C2 T 4P3+2P4)+1P2-1P3

[0137] Δ1C3=1C2(4C2) T 4C3(1C3) T

[0138] Among them, 1, 2, 3, and 4 represent a coordinate system respectively. Optionally, 1 represents the motion capture camera coordinate system, 2 represents the calibration plate coordinate system, 3 represents the rigid body coordinate system, and 4 represents the camera coordinate system to be calibrated.

[0139] Among them, a string in the form of uCv represents the posture of the object corresponding to the v coordinate system relative to the object corresponding to the u coordinate system, and a string in the form of uPv represents the position of the object corresponding to the v coordinate system relative to the object corresponding to the u coordinate system. Optionally, the object corresponding to the coordinate system 1 is a motion capture camera, the object corresponding to the coordinate system 2 is a calibration plate, the object corresponding to the coordinate system 3 is a rigid body, and the object corresponding to the coordinate system 4 is a camera to be calibrated. In this way, 1C2 represents the posture of the calibration plate relative to the motion capture camera, 4C2 represents the posture of the calibration plate relative to the camera to be calibrated, 4P3 represents the position of the rigid body relative to the camera to be calibrated, and 2P4 represents the position of the camera to be calibrated relative to the calibration plate.

[0140] By finding the solution when Δ1P3 and Δ1C3 are minimized, the optimal 4P2 (i.e., the optimized position in step 1605) and the optimal 4C2 (i.e., the optimized posture in step 1607) can be determined.

[0141] In an embodiment of the present disclosure, after obtaining the first reference pose and the fourth calculated pose, the position and posture in the first calculated pose can be optimized based on the first reference pose and the fourth calculated pose, respectively, so as to achieve the overall optimization of the first calculated pose efficiently and quickly.

[0142] Any method for implementing camera calibration provided in the embodiments of the present disclosure may be executed by any appropriate device with data processing capabilities, including but not limited to: a terminal device and a server, etc. Alternatively, any method for implementing camera calibration provided in the embodiments of the present disclosure may be executed by a processor, such as the processor executing any method for implementing camera calibration mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. This will not be described in detail below.

[0143] Exemplary Devices

[0144] Figure 5 is a schematic structural diagram of a device for implementing camera calibration provided by an exemplary embodiment of the present disclosure, Figure 5 The device shown includes a first determination module 520 , a second determination module 540 , a third determination module 560 and a calibration module 580 .

[0145] A first determination module 520 is used to determine a second calculated pose of the calibration plate relative to the motion capture camera and a third calculated pose of the camera to be calibrated relative to the rigid body based on a first reference pose of the rigid body relative to the motion capture camera and a first calculated pose of the calibration plate relative to the camera to be calibrated, wherein the camera to be calibrated is installed on the rigid body, and the rigid body can drive the camera to be calibrated to move;

[0146] A second determination module 540, configured to determine a fourth calculated pose of the rigid body relative to the motion capture camera based on the first calculated pose, the second calculated pose, and the third calculated pose;

[0147] A third determination module 560 is used to optimize the first calculated pose by using a first error between the fourth calculated pose and the first reference pose, so as to determine an optimized pose of the calibration plate relative to the camera to be calibrated;

[0148] The calibration module 580 is used to calibrate the camera to be calibrated based on the optimized posture.

[0149] In an alternative example, Figure 6 As shown, the device also includes:

[0150] The control module 502 is used to control multiple cameras in the motion capture camera to respectively capture a first image sequence of the rigid body and control the camera to be calibrated to capture a second image sequence of the calibration plate before determining a second calculated pose of the calibration plate relative to the motion capture camera and a third calculated pose of the camera to be calibrated relative to the rigid body based on a first reference pose of the rigid body relative to the motion capture camera and a first calculated pose of the calibration plate relative to the camera to be calibrated. In the process of the rigid body driving the camera to be calibrated to move, the multiple cameras in the motion capture camera are controlled to respectively capture a first image sequence of the rigid body and the camera to be calibrated is controlled to capture a second image sequence of the calibration plate.

[0151] A first acquisition module 504 is used to acquire a first pose sequence based on first image sequences corresponding to each of the plurality of cameras in the motion capture camera, wherein the first reference pose is determined based on the first pose sequence;

[0152] The second acquisition module 506 is used to acquire a second posture sequence based on the second image sequence, and the first calculated posture is determined based on the second posture sequence.

[0153] In an optional example, the first image sequence is obtained by collecting images of a rigid body at a first frequency, and the second image sequence is obtained by collecting images of a calibration plate at a second frequency, the second frequency is different from the first frequency, and the first pose sequence corresponds to the first frequency, and the second pose sequence corresponds to the second frequency;

[0154] like Figure 7 As shown, the device also includes:

[0155] A fourth determination module 508, configured to determine a first time difference between the second posture sequence and the first posture sequence;

[0156] The time synchronization module 510 is used to perform a first time synchronization on the first pose sequence and the second pose sequence based on the first time difference. The first reference pose is determined based on the first pose sequence after the first time synchronization. The first calculated pose is determined based on the second pose sequence after the first time synchronization. The first calculated pose is aligned in time with the first reference pose.

[0157] In an optional example, the fourth determining module 508 includes:

[0158] A first extraction submodule, used for extracting a first posture sequence from the first posture sequence;

[0159] A second extraction submodule, used for extracting a second posture sequence from the second posture sequence;

[0160] A processing submodule, used for taking the posture sequence with a higher corresponding frequency in the first posture sequence and the second posture sequence as a reference posture sequence, and performing interpolation processing on the other posture sequence in the first posture sequence and the second posture sequence to obtain an interpolated posture sequence, wherein the frequency corresponding to the interpolated posture sequence is the same as that of the reference posture sequence;

[0161] The determination submodule is used to determine a first time difference between the second posture sequence and the first posture sequence based on the reference posture sequence and the interpolation posture sequence.

[0162] In an optional example, determining a submodule includes:

[0163] A first processing unit, used for performing differential processing on the reference posture sequence to obtain a first angular velocity sequence;

[0164] a second processing unit, configured to perform differentiation processing on the interpolated attitude sequence to obtain a second angular velocity sequence;

[0165] A first determining unit, configured to determine a second time difference between the first angular velocity sequence and the second angular velocity sequence by using a cross-correlation algorithm;

[0166] The second determining unit is used to determine a first time difference between the second posture sequence and the first posture sequence based on the second time difference.

[0167] In an optional example, the second determining unit includes:

[0168] A time synchronization subunit, configured to perform a second time synchronization on the first posture sequence and the second posture sequence based on a second time difference;

[0169] A first determination subunit is used to determine a second reference posture of the rigid body relative to the motion capture camera based on the first posture sequence after second time synchronization;

[0170] A second determination subunit is used to determine a fifth calculated pose of the calibration plate relative to the camera to be calibrated based on the second pose sequence after second time synchronization, and the fifth calculated pose is aligned with the second reference pose in time;

[0171] A third determining subunit is used to determine a sixth calculated pose of the rigid body relative to the motion capture camera based on the second reference pose and the fifth calculated pose;

[0172] a fourth determination subunit, configured to optimize the second time difference by using a second error between the sixth calculated posture and the second reference posture, so as to determine an optimized time difference between the first angular velocity sequence and the second angular velocity sequence;

[0173] The fifth determining subunit is used to determine the optimized time difference as a first time difference between the second posture sequence and the first posture sequence.

[0174] In an alternative example, Figure 8 As shown, the calibration module 580 includes:

[0175] A first extraction submodule 5801, configured to extract a first position and a first posture from a fourth calculated posture;

[0176] A second extraction submodule 5803, configured to extract a second position and a second posture from the first reference posture;

[0177] A first optimization submodule 5805, configured to optimize a third position in the first calculated pose using an error between the first position and the second position, so as to determine an optimized position of the calibration plate relative to the camera to be calibrated;

[0178] The second optimization submodule 5807 is used to optimize the third posture in the first calculated posture by using the error between the first posture and the second posture, so as to determine the optimized posture of the calibration plate relative to the camera to be calibrated.

[0179] Exemplary Electronic Devices

[0180] Below, reference Fig. 9 The electronic device according to the embodiment of the present disclosure is described. The electronic device may be any one or both of the first device and the second device, or a stand-alone device independent of them, and the stand-alone device may communicate with the first device and the second device to receive the collected input signals from them.

[0181] Fig. 9 A block diagram of an electronic device 900 according to an embodiment of the present disclosure is illustrated.

[0182] like Fig. 9 As shown, the electronic device 900 includes one or more processors 901 and a memory 902 .

[0183] The processor 901 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 900 to perform desired functions.

[0184] The memory 902 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 901 may execute the program instructions to implement the methods for implementing camera calibration and / or other desired functions of the various embodiments of the present disclosure described above. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0185] In one example, the electronic device 900 may further include: an input device 903 and an output device 904 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0186] For example, when the electronic device 900 is the first device or the second device, the input device 903 may be a microphone or a microphone array. When the electronic device 900 is a stand-alone device, the input device 903 may be a communication network connector for receiving collected input signals from the first device and the second device.

[0187] In addition, the input device 903 may also include, for example, a keyboard, a mouse, and the like.

[0188] The output device 904 can output various information to the outside, and can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto.

[0189] Of course, to simplify, Fig. 9 Only some of the components related to the present disclosure in the electronic device 900 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 900 may also include any other appropriate components.

[0190] Exemplary computer program products and computer-readable storage media

[0191] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps in the method for implementing camera calibration according to various embodiments of the present disclosure described in the above-mentioned “Exemplary Method” section of this specification.

[0192] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0193] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, and when the computer program instructions are executed by a processor, the processor executes the steps in the method for implementing camera calibration according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0194] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0195] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0196] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0197] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0198] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0199] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0200] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0201] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for implementing camera calibration, comprising: Based on a first reference pose of a rigid body relative to a motion capture camera and a first calculated pose of a calibration plate relative to a camera to be calibrated, determining a second calculated pose of the calibration plate relative to the motion capture camera and a third calculated pose of the camera to be calibrated relative to the rigid body, wherein the camera to be calibrated is mounted on the rigid body, and the rigid body can drive the camera to be calibrated to move; Determining a fourth calculated pose of the rigid body relative to the motion capture camera based on the first calculated pose, the second calculated pose, and the third calculated pose; Optimizing the first calculated pose by using a first error between the fourth calculated pose and the first reference pose to determine an optimized pose of the calibration plate relative to the camera to be calibrated; Based on the optimized posture, the camera to be calibrated is calibrated.

2. The method according to claim 1, wherein: Before determining the second calculated pose of the calibration plate relative to the motion capture camera and the third calculated pose of the camera to be calibrated relative to the rigid body based on the first reference pose of the rigid body relative to the motion capture camera and the first calculated pose of the calibration plate relative to the camera to be calibrated, the method further comprises: In the process of the rigid body driving the camera to be calibrated to move, controlling multiple cameras in the motion capture camera to respectively capture a first image sequence of the rigid body, and controlling the camera to be calibrated to capture a second image sequence of the calibration plate; Based on the first image sequences respectively corresponding to a plurality of cameras in the motion capture cameras, a first pose sequence is acquired, wherein the first reference pose is determined based on the first pose sequence; Based on the second image sequence, a second pose sequence is acquired, and the first calculated pose is determined based on the second pose sequence.

3. The method according to claim 2, wherein: The first image sequence is obtained by collecting images of the rigid body at a first frequency, and the second image sequence is obtained by collecting images of the calibration plate at a second frequency, the second frequency is different from the first frequency, and the first posture sequence corresponds to the first frequency, and the second posture sequence corresponds to the second frequency; The method further comprises: determining a first time difference between the second sequence of postures and the first sequence of postures; Based on the first time difference, the first pose sequence and the second pose sequence are synchronized in the first time, the first reference pose is determined based on the first pose sequence after the first time synchronization, the first calculated pose is determined based on the second pose sequence after the first time synchronization, and the first calculated pose is aligned in time with the first reference pose.

4. The method according to claim 3, wherein: The determining a first time difference between the second posture sequence and the first posture sequence comprises: extracting a first posture sequence from the first posture sequence; extracting a second posture sequence from the second posture sequence; The posture sequence with a higher corresponding frequency in the first posture sequence and the second posture sequence is used as a reference posture sequence, and the other posture sequence in the first posture sequence and the second posture sequence is interpolated to obtain an interpolated posture sequence, wherein the frequency corresponding to the interpolated posture sequence is the same as that of the reference posture sequence; Based on the reference pose sequence and the interpolated pose sequence, a first time difference between the second pose sequence and the first pose sequence is determined.

5. The method according to claim 4, wherein: The determining, based on the reference posture sequence and the interpolated posture sequence, a first time difference between the second posture sequence and the first posture sequence comprises: Performing differentiation processing on the reference posture sequence to obtain a first angular velocity sequence; Performing differentiation processing on the interpolated posture sequence to obtain a second angular velocity sequence; Determine a second time difference between the first angular velocity sequence and the second angular velocity sequence by using a cross-correlation algorithm; Based on the second time difference, a first time difference between the second sequence of poses and the first sequence of poses is determined.

6. The method according to claim 5, wherein: The determining, based on the second time difference, a first time difference between the second posture sequence and the first posture sequence comprises: Based on the second time difference, performing a second time synchronization on the first posture sequence and the second posture sequence; Determining a second reference pose of the rigid body relative to the motion capture camera based on the first pose sequence after second time synchronization; Determining a fifth calculated pose of the calibration plate relative to the camera to be calibrated based on the second pose sequence after second time synchronization, wherein the fifth calculated pose is aligned with the second reference pose in time; Determining a sixth calculated pose of the rigid body relative to the motion capture camera based on the second reference pose and the fifth calculated pose; optimizing the second time difference by using a second error between the sixth calculated posture and the second reference posture to determine an optimized time difference between the first angular velocity sequence and the second angular velocity sequence; The optimized time difference is determined as a first time difference between the second pose sequence and the first pose sequence.

7. The method according to any one of claims 1 to 6, wherein: The optimizing the first calculated pose by using a first error between the fourth calculated pose and the first reference pose to determine an optimized pose of the calibration plate relative to the camera to be calibrated includes: extracting a first position and a first posture from the fourth calculated posture; extracting a second position and a second posture from the first reference posture; Optimizing a third position in the first calculated pose using an error between the first position and the second position to determine an optimized position of the calibration plate relative to the camera to be calibrated; The third posture in the first calculated posture is optimized by using the error between the first posture and the second posture to determine the optimized posture of the calibration plate relative to the camera to be calibrated.

8. A device for implementing camera calibration, comprising: A first determination module is used to determine a second calculated pose of the calibration plate relative to the motion capture camera and a third calculated pose of the camera to be calibrated relative to the rigid body based on a first reference pose of the rigid body relative to the motion capture camera and a first calculated pose of the calibration plate relative to the camera to be calibrated, wherein the camera to be calibrated is installed on the rigid body, and the rigid body can drive the camera to be calibrated to move; a second determination module, configured to determine a fourth calculated pose of the rigid body relative to the motion capture camera based on the first calculated pose, the second calculated pose, and the third calculated pose; A third determination module, configured to optimize the first calculated pose by using a first error between the fourth calculated pose and the first reference pose, so as to determine an optimized pose of the calibration plate relative to the camera to be calibrated; A calibration module is used to calibrate the camera to be calibrated based on the optimized posture.

9. An electronic device, comprising: A memory for storing a computer program product; A processor, configured to execute a computer program product stored in the memory, and when the computer program product is executed, implements the method for camera calibration described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, wherein when the computer program instructions are executed by a processor, the method for realizing camera calibration according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising computer program instructions, wherein when the computer program instructions are executed by a processor, the method for camera calibration according to any one of claims 1 to 7 is implemented.

Images