Method and apparatus for aligning tracking system and camera, device, and storage medium

Abstract

Embodiments of the present disclosure provide a method and apparatus for aligning a tracking system and a camera, a device, and a storage medium. The method comprises: using a camera to acquire the pose of a moving calibration plate in a camera coordinate system; using a tracking system to acquire the poses of the camera and the moving calibration plate in a tracking system coordinate system; and, on the basis of the poses of the calibration plate in the camera coordinate system and the tracking system coordinate system, the pose of the camera in the tracking system coordinate system, and the frame rate of the tracking system, determining a time difference and a frame number difference between the camera and the tracking system to synchronously align the tracking system and the camera. According to the solution, only the calibration plate is introduced, the poses of the calibration plate in the corresponding different coordinate systems and the pose of the camera in the tracking system coordinate system are acquired on the basis of the camera and the tracking system, and the time difference and the frame number difference between the camera and the tracking system are determined, so that the tracking system and the camera can be aligned, thereby improving the universality, convenience, and accuracy of alignment of the camera and the tracking system and reducing the costs of the alignment process.

Description

Method, device, apparatus and storage medium for aligning tracking system and camera Technical Field

[0001] The present disclosure relates to the field of automatic control technology, and in particular to a method, apparatus, device, and storage medium for aligning a tracking system and a camera. Background Art

[0002] Latency is an unavoidable characteristic of any computer system, and Motion-to-Photon (MTP) latency is one of them. It describes the difference between the moment a tracked object starts moving and the moment it appears on the display. This latency can affect the performance of interactive image display programs.

[0003] One current idea is to artificially add a delay to the tracking system so that the user can receive the delayed tracking pose information and image information at the same time. In scenarios where real-time performance is not emphasized and only alignment of pose information and image information is required, this method is very applicable and convenient. However, the current alignment method using this idea is cumbersome and has poor versatility. Summary of the Invention

[0004] The present disclosure provides a method, apparatus, device, and storage medium for aligning a tracking system and a camera.

[0005] According to a first aspect of the present disclosure, a method for aligning a tracking system and a camera is provided, the method comprising:

[0006] Use the camera to obtain the position and posture of the moving calibration plate in the camera coordinate system;

[0007] Use the tracking system to obtain the position and pose of the camera and the moving calibration plate in the tracking system coordinate system;

[0008] Determine the time difference and frame difference between the camera and the tracking system based on the position of the calibration plate in the camera coordinate system and the tracking system coordinate system, the position of the camera in the tracking system coordinate system, and the frame rate of the tracking system;

[0009] Based on the time difference and frame rate difference between the camera and tracking system, the tracking system and camera are synchronized and aligned.

[0010] In some implementations of the first aspect, determining the time difference and frame difference between the camera and the tracking system based on the pose of the calibration plate in the camera coordinate system and the tracking system coordinate system, the pose of the camera in the tracking system coordinate system, and the frame rate of the tracking system includes:

[0011] Determine the average velocity of the calibration plate based on the coordinates of the current frame and the k frames before the current frame in the tracking system coordinate system and the frame rate of the tracking system, where k is a positive integer greater than or equal to 1;

[0012] Determine the time difference between the camera and the tracking system based on the average velocity of the calibration plate, the position of the calibration plate in the camera coordinate system and the tracking system coordinate system, and the position of the camera in the tracking system coordinate system;

[0013] Based on the time difference between the camera and the tracking system and the frame rate of the tracking system, the frame difference between the camera and the tracking system is determined.

[0014] In some implementations of the first aspect, determining the time difference between the camera and the tracking system based on the average velocity of the calibration plate, the pose of the calibration plate in the camera coordinate system and the tracking system coordinate system, and the pose of the camera in the tracking system coordinate system includes:

[0015] Based on the pose of the calibration plate in the camera coordinate system and the pose of the camera in the tracking system coordinate system, the pose of the calibration plate in the camera coordinate system is transformed into the tracking system coordinate system to obtain the pose S1;

[0016] According to the formula , determine the time difference between the camera and the tracking system, where is the time difference between the camera and the tracking system, S2 is the position of the calibration plate obtained by the tracking system in the tracking system coordinate system, is the average speed of the calibration plate;

[0017] Based on the time difference between the camera and the tracking system and the frame rate of the tracking system, the frame difference between the camera and the tracking system is determined, including:

[0018] According to the formula , determine the frame difference between the camera and the tracking system, where is the frame difference between the camera and the tracking system, is the time difference between the camera and the tracking system, is the frame rate of the tracking system.

[0019] In some implementations of the first aspect, synchronously aligning the tracking system and the camera based on a time difference and a frame difference between the camera and the tracking system includes:

[0020] When the difference between the current pose of the calibration plate in the camera coordinate system converted to the pose in the tracking system coordinate system and the current pose of the calibration plate in the compensated tracking system coordinate system is greater than the threshold, increase the k value, determine the new time difference and new frame difference between the camera and the tracking system, and repeat the cycle until the tracking system is compensated based on the latest time difference and the latest frame difference, and the difference between the current pose of the calibration plate in the camera coordinate system converted to the pose in the tracking system coordinate system and the current pose of the calibration plate in the latest compensated tracking system coordinate system is less than or equal to the threshold;

[0021] The tracking system is compensated based on the latest time difference and latest frame difference between the camera and the tracking system, and the tracking system and camera are synchronously aligned. The time difference and frame difference between the current frame and the j-1 frames before the current frame are calculated, and the preset stability conditions are met. Where j is a positive integer greater than or equal to 2.

[0022] In some implementations of the first aspect, when the time difference and frame number difference after latest compensation of the current frame and j-1 frames before the current frame respectively satisfy a preset stability condition, the tracking system is compensated based on the latest time difference and latest frame number difference between the camera and the tracking system, and the tracking system and the camera are synchronously aligned, including:

[0023] Determine the average time difference based on the latest compensated time difference between the current frame and the j-1 frames before the current frame;

[0024] Determine an average frame number difference based on the latest compensated frame number difference between the current frame and the j-1 frames before the current frame;

[0025] When the average time difference and average frame difference meet the preset stability conditions, the tracking system is compensated based on the latest time difference and latest frame difference between the camera and the tracking system, and the tracking system and the camera are synchronously aligned.

[0026] In some implementations of the first aspect, using the camera to obtain the position and pose of the moving calibration plate in the camera coordinate system includes:

[0027] Obtain the size information of the calibration plate, and use the camera to obtain the target frame image data corresponding to the moving complete calibration plate;

[0028] Based on the target frame image data, determine the corner point positions of the calibration plate in the camera coordinate system;

[0029] Based on the size information of the calibration plate and the corner position of the calibration plate in the camera coordinate system, the n-point perspective positioning PnP algorithm is used to determine the position and posture of the calibration plate in the camera coordinate system.

[0030] In some implementations of the first aspect, the preset stability condition includes that the average time difference is less than a preset stable time difference threshold and the average frame number difference is less than a preset stable frame number difference threshold.

[0031] According to a second aspect of the present disclosure, there is provided a device for aligning a tracking system and a camera, the device comprising:

[0032] The pose acquisition module is used to use the camera to obtain the pose of the moving calibration plate in the camera coordinate system;

[0033] The posture acquisition module is also used to use the tracking system to obtain the posture of the moving calibration plate in the tracking system coordinate system;

[0034] a processing module, configured to determine the time difference and frame difference between the camera and the tracking system based on the pose of the calibration plate in the camera coordinate system and the tracking system coordinate system, the pose of the camera in the tracking system coordinate system, and the frame rates of the tracking system and the camera;

[0035] The alignment module is used to synchronize the tracking system and the camera based on the time difference and frame difference between the camera and the tracking system.

[0036] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above method when executing the program.

[0037] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method according to the first aspect of the present disclosure and some implementations of the first aspect is implemented.

[0038] The present invention uses a camera to obtain the position and posture of a moving calibration plate in a camera coordinate system; uses a tracking system to obtain the position and posture of the moving calibration plate in the tracking system coordinate system; determines the time difference and frame difference between the camera and the tracking system based on the position and posture of the calibration plate in the camera coordinate system and the tracking system coordinate system, the position and posture of the camera in the tracking system coordinate system, and the frame rate of the tracking system; and aligns the tracking system and the camera synchronously based on the time difference and frame difference between the camera and the tracking system. By introducing a calibration plate, obtaining the position and posture of the calibration plate in corresponding different coordinate systems based on the camera and the tracking system, and the position and posture of the camera in the tracking system coordinate system, and determining the time difference and frame difference between the camera and the tracking system, the tracking system and the camera can be aligned, thereby improving the versatility, convenience, and accuracy of the alignment of the camera and the tracking system, and reducing the cost of the alignment process.

[0039] It should be understood that the contents described in the Summary of the Invention section are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are provided for a better understanding of the present disclosure and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, among which:

[0041] FIG1 is a schematic diagram showing a flow chart of a tracking system and a camera alignment method according to an embodiment of the present disclosure;

[0042] FIG2 is a schematic flow chart showing another tracking system and camera alignment method according to an embodiment of the present disclosure;

[0043] FIG3 shows a schematic diagram of a calibration plate according to an embodiment of the present disclosure;

[0044] FIG4 is a schematic diagram showing a process of determining the position and posture of a calibration plate in a camera coordinate system using a perspective-n-point (PnP) algorithm according to an embodiment of the present disclosure;

[0045] FIG5 shows a block diagram of a tracking system and a camera alignment apparatus according to an embodiment of the present disclosure;

[0046] FIG6 illustrates a block diagram of an exemplary electronic device capable of implementing embodiments of the present disclosure. DETAILED DESCRIPTION

[0047] To make the purpose, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present disclosure.

[0048] In this document, the term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0049] In the present disclosure, a camera is used to obtain the position and posture of a moving calibration plate in a camera coordinate system; a tracking system is used to obtain the position and posture of a moving calibration plate in a tracking system coordinate system; the time difference and frame difference between the camera and the tracking system are determined based on the position and posture of the calibration plate in the camera coordinate system and the tracking system coordinate system, the position and posture of the camera in the tracking system coordinate system, and the frame rate of the tracking system; and the tracking system and the camera are synchronously aligned based on the time difference and frame difference between the camera and the tracking system. By introducing a calibration plate, obtaining the position and posture of the calibration plate in corresponding different coordinate systems based on the camera and the tracking system, the position and posture of the camera in the tracking system coordinate system, and determining the time difference and frame difference between the camera and the tracking system, the tracking system and the camera can be aligned, thereby improving the versatility, convenience, and accuracy of the alignment of the camera and the tracking system, and reducing the cost of the alignment process.

[0050] FIG1 shows a flow chart of a method for aligning a tracking system and a camera according to an embodiment of the present disclosure. The method may be applied to a processor. As shown in FIG1 , the method 100 for aligning a tracking system and a camera may include:

[0051] S101, using a camera to obtain the position and posture of the moving calibration plate in the camera coordinate system;

[0052] S102, using a tracking system to obtain the position and posture of the moving calibration plate in the tracking system coordinate system;

[0053] S103, determining the time difference and frame difference between the camera and the tracking system based on the position of the calibration plate in the camera coordinate system and the tracking system coordinate system, the position of the camera in the tracking system coordinate system, and the frame rate of the tracking system;

[0054] S104 , based on the time difference and frame number difference between the camera and the tracking system, synchronously aligning the tracking system and the camera.

[0055] In the process of S101-S104, by introducing a calibration plate, the position and posture of the calibration plate in the corresponding different coordinate systems are obtained based on the camera and tracking system, and combined with the position and posture of the camera in the tracking system coordinate system, the time difference and frame difference between the camera and the tracking system are determined. Then, the tracking system and the camera can be aligned, which improves the versatility, convenience and accuracy of the alignment of the camera and tracking system and reduces the cost of the alignment process.

[0056] FIG2 shows a flowchart of another method for aligning a tracking system and a camera according to an embodiment of the present disclosure. The processes S101 to S104 will be further described in conjunction with the method 200 for aligning a tracking system and a camera in FIG2 .

[0057] It should be noted that in the process of moving the calibration plate, it is necessary to ensure that the movement process can be captured by the camera and tracking system to ensure that the position and posture of the moving calibration plate in the camera coordinate system and the tracking system coordinate system can be obtained.

[0058] In actual applications, in order to accurately determine the time difference and frame difference between the camera and the tracking system, in some embodiments, the above S103 may determine the time difference and frame difference between the camera and the tracking system based on the position of the calibration plate in the camera coordinate system and the tracking system coordinate system, the position of the camera in the tracking system coordinate system, and the frame rate of the tracking system. Specifically, the following steps may be performed:

[0059] Determine the average velocity of the calibration plate based on the coordinates of the current frame and the k frames before the current frame in the tracking system coordinate system and the frame rate of the tracking system, where k can be a positive integer greater than or equal to 1;

[0060] Determine the time difference between the camera and the tracking system based on the average velocity of the calibration plate, the position of the calibration plate in the camera coordinate system and the tracking system coordinate system, and the position of the camera in the tracking system coordinate system;

[0061] Based on the time difference between the camera and the tracking system and the frame rate of the tracking system, the frame difference between the camera and the tracking system is determined.

[0062] In the above embodiment, the average speed of the calibration plate is first determined by the coordinates of the current frame and the k frames before the current frame in the tracking system coordinate system and the frame rate of the tracking system. Then, based on the average speed of the calibration plate and the position of the calibration plate in the camera coordinate system and the tracking system coordinate system, and the position of the camera in the tracking system coordinate system, the time difference between the camera and the tracking system is determined. Finally, based on the time difference between the camera and the tracking system and the frame rate of the tracking system, the frame difference between the camera and the tracking system is accurately determined.

[0063] In some embodiments, in order to accurately calculate the time difference and frame difference, the above-mentioned method of determining the time difference between the camera and the tracking system based on the average speed of the calibration plate, the position of the calibration plate in the camera coordinate system and the tracking system coordinate system, and the position of the camera in the tracking system coordinate system may specifically include:

[0064] Based on the pose of the calibration plate in the camera coordinate system and the pose of the camera in the tracking system coordinate system, the pose of the calibration plate in the camera coordinate system is transformed into the tracking system coordinate system to obtain the pose S1;

[0065] According to the formula , determine the time difference between the camera and the tracking system, where is the time difference between the camera and the tracking system, S2 is the position of the calibration plate obtained by the tracking system in the tracking system coordinate system, is the average speed of the calibration plate;

[0066] Determining the frame difference between the camera and the tracking system based on the time difference between the camera and the tracking system and the frame rate of the tracking system may include:

[0067] According to the formula , determine the frame difference between the camera and the tracking system, where is the frame difference between the camera and the tracking system, is the time difference between the camera and the tracking system, is the frame rate of the tracking system.

[0068] In the above embodiment, various parameters are brought into the equation using a specific formula, thereby accurately calculating the time difference and the frame difference, so that the tracking system can be accurately compensated subsequently.

[0069] In some embodiments, considering that the time difference and frame rate difference between the camera and the tracking system cannot be accurately determined, the process of aligning the tracking system and the camera based on the time difference and frame rate difference between the camera and the tracking system can be a cyclic tuning process, which specifically may include:

[0070] When the difference between the current pose of the calibration plate in the camera coordinate system converted to the pose in the tracking system coordinate system and the current pose of the calibration plate in the compensated tracking system coordinate system is greater than the threshold, increase the k value, determine the new time difference and new frame difference between the camera and the tracking system, and repeat the cycle until the tracking system is compensated based on the latest time difference and the latest frame difference, and the difference between the current pose of the calibration plate in the camera coordinate system converted to the pose in the tracking system coordinate system and the current pose of the calibration plate in the latest compensated tracking system coordinate system is less than or equal to the threshold;

[0071] Calculate the latest compensated time difference and frame number difference between the current frame and the j-1 frames before the current frame, and if they meet the preset stability conditions, compensate the tracking system based on the latest time difference and frame number difference between the camera and the tracking system, and align the tracking system and the camera. j can be a positive integer greater than or equal to 2.

[0072] In the above embodiment, when the difference between the current pose of the calibration plate in the camera coordinate system converted to the pose in the tracking system coordinate system and the current pose of the calibration plate in the compensated tracking system coordinate system is greater than a threshold, the k value is increased, and the new time difference and new frame difference between the camera and the tracking system are recalculated in a continuous cycle until the tracking system is compensated based on the latest time difference and the latest frame difference, and the pose difference is less than or equal to the threshold, thereby achieving accurate compensation of the tracking system based on the latest time difference and the latest frame difference between the camera and the tracking system.

[0073] In some embodiments, when the time difference and frame number difference after the latest compensation of the current frame and the j-1 frames before the current frame respectively meet the preset stability condition, the tracking system is compensated based on the latest time difference and the latest frame number difference between the camera and the tracking system, and the tracking system and the camera are synchronously aligned, including:

[0074] Determine the average time difference based on the latest compensated time difference between the current frame and the j-1 frames before the current frame;

[0075] Determine an average frame number difference based on the latest compensated frame number difference between the current frame and the j-1 frames before the current frame;

[0076] When the average time difference and average frame difference meet the preset stability conditions, the tracking system is compensated based on the latest time difference and latest frame difference between the camera and the tracking system, and the tracking system and the camera are synchronously aligned.

[0077] Specifically, taking the current frame i as an example, after calculating the time difference Δt and frame number difference n of the i-th frame, the time difference and frame number difference of the i-j+1th frame, the i-j+2th frame, ..., the i-th frame (a total of j frames) are combined and averaged to obtain the final time difference and frame number difference. If the preset stability conditions are met, the tracking system is compensated based on the latest time difference and latest frame number difference between the camera and the tracking system, and the tracking system and the camera are synchronously aligned.

[0078] In the above embodiment, the average time difference and average frame difference after compensation of the tracking system are compared with the preset stability conditions. When the preset stability conditions are met, the tracking system and the camera are accurately aligned based on the latest time difference and latest frame difference between the camera and the tracking system.

[0079] It should also be noted that if the average time difference and the average frame number difference do not meet the preset stability conditions, the process may proceed to the i+1th frame and repeat the above-mentioned method of aligning the tracking system and the camera to continue alignment.

[0080] In some embodiments, using the camera to obtain the position and posture of the moving calibration plate in the camera coordinate system may include:

[0081] Obtain the size information of the calibration plate, and use the camera to obtain the target frame image data corresponding to the moving complete calibration plate;

[0082] Based on the target frame image data, determine the corner point positions of the calibration plate in the camera coordinate system;

[0083] Based on the size information of the calibration plate and the position of the corner points of the calibration plate in the camera coordinate system, the perspective-n-point (PnP) algorithm is used to determine the position and orientation of the calibration plate in the camera coordinate system.

[0084] In the above embodiment, by identifying the corner point positions of the calibration plate in the target frame image data in the camera coordinate system, and based on the size information of the calibration plate, the PnP algorithm is used to accurately determine the position and posture of the calibration plate in the camera coordinate system, so that the tracking system and the camera can be accurately aligned subsequently.

[0085] In some embodiments, the calibration plate may be a checkerboard as shown in FIG3 , which is used to calibrate the camera's internal parameters to align the tracking system and the camera. Depending on the requirements, the size of the checkerboard and the internal checkerboard composition may also vary. The present invention has no restrictions on the size of the checkerboard and the shape of the internal pattern.

[0086] Specifically, with reference to the checkerboard shown in FIG3 , after capturing the complete checkerboard, the camera detects the corner points on the checkerboard using a corner detection algorithm, thereby obtaining the coordinates of the checkerboard in the camera coordinate system. It should be noted that the corner detection algorithm can be run on an independent host computer. If the camera hardware permits, it can also be burned into the camera, allowing the algorithm to run in the camera without relying on the computer, thereby reducing the consumption of computer computing resources.

[0087] A corner is the intersection of two adjacent edges in a graphic. It is a simple but effective feature and is therefore frequently used in feature detection. Common corner detection algorithms include the Harris corner detection method, the Shi-Tomasi corner detection method, and the Features from accelerated segment test (FAST) corner detection algorithm.

[0088] The above-mentioned process of accurately determining the pose of the calibration plate in the camera coordinate system using the PnP algorithm can be specifically shown in Figure 4. The determination process is further described in combination with the method 400 for determining the pose of the calibration plate in the camera coordinate system using the PnP algorithm shown in Figure 4. In a certain frame, the camera first obtains the captured image, and then identifies whether there is a checkerboard in the image. If so, the corner position of the checkerboard is identified, and the scene is converted into a PnP problem to solve, and then the pose of the checkerboard in the camera coordinate system is solved.

[0089] The PnP algorithm, that is, the method for solving the PnP problem is introduced as follows:

[0090] Given the three-dimensional coordinates P1, P2, ..., Pi, ..., Pn of n points in the world coordinate system, their two-dimensional image coordinates p1, p2, ..., Pi, ..., pn in the camera coordinate system, and the camera's intrinsic parameter matrix K, find the coordinates of the camera coordinate system relative to the world coordinate system. The intrinsic parameter matrix K is defined as follows:

[0091]

[0092] in, is the intrinsic parameter of the camera, which is obtained through camera intrinsic parameter calibration.

[0093] Commonly used methods for solving the PnP problem include direct linear transformation (DLT), efficient n-point perspective positioning (EPnP) algorithm, and bundle adjustment.

[0094] In this scenario, the world coordinate system of the PnP problem is set to the checkerboard coordinate system. Since the size of the checkerboard itself is known, the three-dimensional coordinates of the corner points in the checkerboard coordinate system are known, and the two-dimensional image coordinates of the corner points can be obtained through the corner point recognition algorithm of the camera image. The intrinsic parameter matrix of the camera can be obtained through the camera intrinsic parameter calibration. Therefore, the PnP problem in this scenario is solvable, and the pose of the camera coordinate system relative to the checkerboard coordinate system will be obtained, and the pose of the checkerboard in the camera coordinate system can also be obtained.

[0095] At this point, the positions of the checkerboard in the tracking system and the camera are already known. Since the camera can be tracked by the tracking system, the positions of the checkerboard in the tracking system and the camera coordinate system can be unified into the same coordinate system. The same coordinate system can be the tracking system, and can also be adjusted accordingly according to actual conditions.

[0096] In some embodiments, the preset stability condition includes that the average time difference is less than a preset stable time difference threshold and the average frame number difference is less than a preset stable frame number difference threshold.

[0097] In the above embodiment, by setting a preset stabilization condition, the tracking system and the camera can be accurately aligned.

[0098] In the above embodiment, the tracking system may include a tracker and a tracker. The tracker can be tracked by the tracker. The tracker, the checkerboard, and the camera are rigidly connected. The tracker can obtain the position and posture of the checkerboard and the camera in the coordinate system of the tracking system by tracking the locator. Since the delay of the tracking system itself is extremely small, it can be ignored.

[0099] During surgery using an endoscope, taking the endoscope as the camera in the above embodiment, and the actual use process of using the tracking system to collect data as an example, the endoscope is used to collect data for neural network training in conjunction with the tracking system. Due to the MTP delay of the endoscope itself, there will be a considerable time interval between the two steps of collecting and displaying the image, resulting in the object pose in the image it outputs always lagging behind the object pose tracked by the tracking system. Therefore, it is necessary to align the image output by the endoscope with the tracking system; however, the hardware of the endoscope itself is difficult to change, and the object pose output by the tracking system can only be manually delayed to synchronize it with the object pose in the image; since there is no real-time requirement in this scenario, only the tracking system and the endoscope are required to be synchronized, the solution disclosed in this disclosure is used to align the tracking system and the endoscope, which can ensure the universality, convenience and accuracy of the alignment of the endoscope and the tracking system, and reduce the cost of the alignment process.

[0100] The tracking system consists of a tracker and a tracker. The tracker is rigidly connected to the endoscope and the chessboard. The tracker tracks the tracker through infrared light. The positional relationship between the tracker and the corresponding object has been calibrated before, so after the tracker is tracked, the corresponding object can also be tracked.

[0101] The endoscope is connected to a computer. After receiving the image captured by the endoscope, the computer runs the tracking system and camera alignment method of the present invention to obtain the pose of the chessboard in the endoscope coordinate system, namely:

[0102] First, the image captured by the endoscope is acquired. Then, it is determined whether there is a checkerboard in the image. If so, the corner positions of the checkerboard are identified, and the scene is converted into a PnP problem. The pose of the checkerboard in the endoscope coordinate system is then solved.

[0103] After obtaining the pose of the chessboard in the endoscope coordinate system and the tracking system coordinate system, they are unified into the same coordinate system. Then, the number of delayed frames and the delay time are calculated based on the system frame rate and the average movement speed of the chessboard. The correction result is substituted into the program to detect whether the difference between the pose of the chessboard in the camera and the pose in the tracking system is small enough. If it is small enough, the result is output; otherwise, the value of the average movement speed of the chessboard is changed, and the delay time and number of frames are continued to be iteratively calculated.

[0104] After calculating the delay time and frame number of the current frame, the time difference and frame number difference of the previous j frames are combined and averaged to obtain the final time difference and frame number difference. If the result meets the preset stability conditions, the result is output for correction to align the tracking system and camera; otherwise, it goes to the next frame, continues to move the checkerboard and repeats the above steps.

[0105] The above is an introduction to the method embodiment. The following further illustrates the disclosed solution through an apparatus embodiment.

[0106] FIG5 shows a block diagram of a tracking system and a camera alignment apparatus according to an embodiment of the present disclosure.

[0107] As shown in FIG5 , the apparatus 500 for aligning a tracking system and a camera may include:

[0108] A posture acquisition module 501 is used to use a camera to acquire the posture of the moving calibration plate in the camera coordinate system;

[0109] The posture acquisition module 501 is further used to use the tracking system to obtain the posture of the moving calibration plate in the tracking system coordinate system;

[0110] Processing module 502, for determining the time difference and frame difference between the camera and the tracking system based on the position of the calibration plate in the camera coordinate system and the tracking system coordinate system, the position of the camera in the tracking system coordinate system, and the frame rates of the tracking system and the camera;

[0111] The alignment module 503 is used to synchronize the tracking system with the camera based on the time difference and frame difference between the camera and the tracking system.

[0112] In some embodiments, the processing module 502 can also be used to determine the average speed of the calibration plate based on the coordinates of the current frame and the k-th frames before the current frame in the tracking system coordinate system and the frame rate of the tracking system, where k is a positive integer greater than or equal to 1; determine the time difference between the camera and the tracking system based on the average speed of the calibration plate, the position of the calibration plate in the camera coordinate system and the tracking system coordinate system, and the position of the camera in the tracking system coordinate system; determine the frame number difference between the camera and the tracking system based on the time difference between the camera and the tracking system and the frame rate of the tracking system.

[0113] In some embodiments, the processing module 502 may also be used to transform the pose of the calibration plate in the camera coordinate system into the tracking system coordinate system based on the pose of the calibration plate in the camera coordinate system and the pose of the camera in the tracking system coordinate system to obtain the pose S1; according to the formula , determine the time difference between the camera and the tracking system, where is the time difference between the camera and the tracking system, S2 is the position of the calibration plate obtained by the tracking system in the tracking system coordinate system, is the average speed of the calibration plate; according to the formula , determine the frame difference between the camera and the tracking system, where is the frame difference between the camera and the tracking system, is the time difference between the camera and the tracking system, is the frame rate of the tracking system.

[0114] In some embodiments, the alignment module 503 can also be used to increase the k value when the difference between the current pose of the calibration plate in the camera coordinate system converted to the pose in the tracking system coordinate system and the current pose of the calibration plate in the compensated tracking system coordinate system is greater than a threshold, determine the new time difference and the new frame difference between the camera and the tracking system, and continuously loop until the tracking system is compensated based on the latest time difference and the latest frame difference, and the difference between the current pose of the calibration plate in the camera coordinate system converted to the pose in the tracking system coordinate system and the current pose of the calibration plate in the latest compensated tracking system coordinate system is less than or equal to the threshold; calculate the latest compensated time difference and frame difference of each frame in the current frame and the j-1 frames before the current frame respectively, and when the preset stability condition is satisfied, compensate the tracking system based on the latest time difference and the latest frame difference between the camera and the tracking system, and synchronously align the tracking system and the camera, where j is a positive integer greater than or equal to 2.

[0115] In some embodiments, the alignment module 503 can also be used to determine the average time difference based on the latest compensated time difference between the current frame and the j-1 frames before the current frame; determine the average frame difference based on the latest compensated frame difference between the current frame and the j-1 frames before the current frame; and when the average time difference and the average frame difference meet preset stability conditions, compensate the tracking system based on the latest time difference and the latest frame difference between the camera and the tracking system, and synchronize the tracking system and the camera.

[0116] In some embodiments, the posture acquisition module 501 can also be used to obtain the size information of the calibration plate, and use the camera to obtain the target frame image data corresponding to the moving complete calibration plate; based on the target frame image data, determine the corner point position of the calibration plate in the camera coordinate system; based on the size information of the calibration plate and the corner point position of the calibration plate in the camera coordinate system, use the PnP algorithm to determine the posture of the calibration plate in the camera coordinate system.

[0117] In some embodiments, the preset stability condition includes that the average time difference is less than a preset stable time difference threshold and the average frame number difference is less than a preset stable frame number difference threshold.

[0118] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0119] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0120] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0121] FIG6 shows a block diagram of an exemplary electronic device capable of implementing an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0122] Device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. RAM 603 may also store various programs and data required for the operation of device 600. Computing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to bus 604.

[0123] Various components in device 600 are connected to I / O interface 605, including an input unit 606, such as a keyboard, mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a magnetic disk, optical disk, etc.; and a communication unit 609, such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows device 600 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0124] Computing unit 601 may be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Computing unit 601 performs the various methods and processes described above, such as method 100. For example, in some embodiments, method 100 may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed onto device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by computing unit 601, one or more steps of method 100 described above may be performed.

[0125] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0126] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0127] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0128] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0129] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0130] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0131] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions of this disclosure can be achieved, and this document is not limited here.

[0132] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for aligning a tracking system and a camera, characterized in that: The method comprises: Use the camera to obtain the position and posture of the moving calibration plate in the camera coordinate system; Use the tracking system to obtain the position and pose of the camera and the moving calibration plate in the tracking system coordinate system; According to the position and posture of the calibration plate in the camera coordinate system and the tracking system coordinate system, the position and posture of the camera in the tracking system coordinate system, and the frame rate of the tracking system, the time difference and frame number difference between the camera and the tracking system are determined; Based on the time difference and frame rate difference between the camera and the tracking system, the tracking system and the camera are synchronized and aligned.

2. The method according to claim 1, characterized in that Determining the time difference and frame number difference between the camera and the tracking system according to the position and posture of the calibration plate in the camera coordinate system and the tracking system coordinate system, the position and posture of the camera in the tracking system coordinate system, and the frame rate of the tracking system includes: Determine the average speed of the calibration plate according to the coordinates of the current frame and the kth frame before the current frame in the tracking system coordinate system and the frame rate of the tracking system, where k is a positive integer greater than or equal to 1; Determine the time difference between the camera and the tracking system based on the average speed of the calibration plate, the position and posture of the calibration plate in the camera coordinate system and the tracking system coordinate system, and the position and posture of the camera in the tracking system coordinate system; Based on the time difference between the camera and the tracking system and the frame rate of the tracking system, the frame number difference between the camera and the tracking system is determined.

3. The method according to claim 2, characterized in that The method of determining the time difference between the camera and the tracking system based on the average speed of the calibration plate, the position and posture of the calibration plate in the camera coordinate system and the tracking system coordinate system, and the position and posture of the camera in the tracking system coordinate system includes: Based on the pose of the calibration plate in the camera coordinate system and the pose of the camera in the tracking system coordinate system, the pose of the calibration plate in the camera coordinate system is transformed into the tracking system coordinate system to obtain the pose S1; according to the formula , determine the time difference between the camera and the tracking system, where is the time difference between the camera and the tracking system, S2 is the position and posture of the calibration plate obtained by the tracking system in the tracking system coordinate system, is the average speed of the calibration plate; The determining the frame number difference between the camera and the tracking system based on the time difference between the camera and the tracking system and the frame rate of the tracking system includes: According to the formula , determine the frame difference between the camera and the tracking system, where is the frame difference between the camera and the tracking system, is the time difference between the camera and the tracking system, is the frame rate of the tracking system.

4. The method according to claim 2, characterized in that The step of synchronizing the tracking system and the camera based on the time difference and frame number difference between the camera and the tracking system includes: When the difference between the current pose of the calibration plate in the camera coordinate system converted to the pose in the tracking system coordinate system and the current pose of the calibration plate in the compensated tracking system coordinate system is greater than the threshold, increase the k value, determine the new time difference and new frame difference between the camera and the tracking system, and repeat the process until the tracking system is compensated based on the latest time difference and the latest frame difference, and the difference between the current pose of the calibration plate in the camera coordinate system converted to the pose in the tracking system coordinate system and the current pose of the calibration plate in the latest compensated tracking system coordinate system is less than or equal to the threshold; The tracking system is compensated based on the latest time difference and latest frame difference between the camera and the tracking system, and the tracking system and the camera are synchronously aligned. The time difference and frame number difference of each frame in the current frame and the j-1 frames before the current frame are calculated respectively. When the preset stability conditions are met, the tracking system is compensated based on the latest time difference and latest frame number difference between the camera and the tracking system, and the tracking system and the camera are synchronously aligned, where j is a positive integer greater than or equal to 2.

5. The method according to claim 4, characterized in that The method of respectively calculating the time difference and frame number difference of each frame after the latest compensation in the current frame and the j-1 frames before the current frame, and compensating the tracking system based on the latest time difference and the latest frame number difference between the camera and the tracking system, and aligning the tracking system and the camera synchronously, includes: Determine an average time difference based on the latest compensated time difference in the current frame and j-1 frames before the current frame; Determine an average frame number difference based on the latest compensated frame number difference between the current frame and j-1 frames before the current frame; When the average time difference and the average frame number difference meet a preset stability condition, the tracking system is compensated based on the latest time difference and the latest frame number difference between the camera and the tracking system, and the tracking system and the camera are synchronously aligned.

6. The method according to claim 1, characterized in that Use the camera to obtain the position and posture of the moving calibration plate in the camera coordinate system, including: Obtain the size information of the calibration plate, and use the camera to obtain the target frame image data corresponding to the moving complete calibration plate; Based on the target frame image data, determine the corner point position of the calibration plate in the camera coordinate system; Based on the size information of the calibration plate and the corner positions of the calibration plate in the camera coordinate system, the n-point perspective positioning PnP algorithm is used to determine the position and posture of the calibration plate in the camera coordinate system.

7. The method according to claim 4, characterized in that The preset stability condition includes that the average time difference is less than a preset stable time difference threshold and the average frame number difference is less than a preset stable frame number difference threshold.

8. A device for aligning a tracking system and a camera, characterized in that: The device comprises: A posture acquisition module, used to use a camera to acquire the posture of the moving calibration plate in the camera coordinate system; The position and posture acquisition module is further used to use the tracking system to acquire the position and posture of the moving calibration plate in the tracking system coordinate system; A processing module, used to determine the time difference and frame difference between the camera and the tracking system according to the position and posture of the calibration plate in the camera coordinate system and the tracking system coordinate system, the position and posture of the camera in the tracking system coordinate system, and the frame rates of the tracking system and the camera; The alignment module is used to synchronize the tracking system and the camera based on the time difference and frame difference between the camera and the tracking system.

9. An electronic device, characterized in that: include: at least one processor; and a memory communicatively coupled to the at least one processor; It is characterized in that the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

Images