Control method, tracking system, and non-transitory computer-readable medium

Abstract

A control method includes the following steps: Capturing at least one image involving a calibration chart with a camera, wherein a first trackable device entity is attached to the camera, and a second trackable device entity is attached to the calibration chart. Tracking the first trackable device attached to the camera with a tracking base station to generate a first rotation transformation matrix, and tracking the second trackable device attached to the calibration chart with the tracking base station to generate a second rotation transformation matrix. Generating a third rotation transformation matrix based on the calibration chart appearing in the at least one image. Calculating a fourth rotation transformation matrix between the camera coordinate system and the first trackable device based on the first, second, and third rotation transformation matrices, wherein the fourth rotation transformation matrix is ​​used to track the camera. This allows for accurate tracking and positioning of the camera, making the objects captured by the camera appear more realistic when combined with a virtual scene.

Description

Technical Field

[0001] This disclosure relates to a tracking system and control method, and more particularly to a tracking system capable of accurately and efficiently tracking a camera. Background Technology

[0002] Recently, various virtual reality (VR), augmented reality (AR), substitutional reality (SR), and / or mixed reality (MR) devices have been developed to provide users with immersive experiences. When a user wears a head-mounted display (HMD) device, their field of vision is covered by immersive content displayed on the HMD. This immersive content displays virtual backgrounds and objects within the immersive scene.

[0003] To create vivid and immersive content, one approach is to film real actors, vehicles, or animals, and then blend these real objects with virtual objects or backgrounds within a virtual scene. To achieve this blending, it's crucial to accurately track the pose data (position and rotation data) of the real objects. Otherwise, these real objects will be placed in the wrong position or along the wrong axis within the immersive scene. Summary of the Invention

[0004] One embodiment of this disclosure discloses a control method comprising the following steps: capturing at least one image involving a calibration chart with a camera, wherein a first trackable device entity is attached to the camera and a second trackable device entity is attached to the calibration chart; tracking the first trackable device and the second trackable device with a tracking base station to generate a first rotation transformation matrix between the first trackable device and the tracking base station, and generating a second rotation transformation matrix between the second trackable device and the tracking base station; generating a third rotation transformation matrix between a camera coordinate system of the camera and the calibration chart based on the calibration chart appearing in the at least one image; and calculating a fourth rotation transformation matrix between the camera coordinate system and the first trackable device based on the first rotation transformation matrix, the second rotation transformation matrix, and the third rotation transformation matrix, wherein the fourth rotation transformation matrix is ​​used for tracking the camera.

[0005] In some embodiments, the calibration chart includes a feature pattern and a vehicle socket, the second trackable device entity is attached to the vehicle socket and has a mechanical configuration relationship relative to the feature pattern, a fifth rotational transformation matrix between the calibration chart and the second trackable device entity is derived from the mechanical configuration relationship, and the calculation of the fourth rotational transformation matrix is ​​further based on the fifth rotational transformation matrix.

[0006] In some embodiments, the fourth rotation transformation matrix is ​​calculated as the product of the third rotation transformation matrix, the fifth rotation transformation matrix, the second rotation transformation matrix, and the first rotation transformation matrix.

[0007] In some embodiments, the origin of the camera coordinate system is located at an optical center of the camera, and the fourth rotation transformation matrix is ​​used to describe a rotational relationship and a positional relationship between the camera coordinate system and the first tracking device.

[0008] In some embodiments, the control method further includes: capturing N images of the calibration chart with the camera, where N is a positive integer greater than 1; and performing a camera geometric calibration based on the calibration chart appearing in the N images to generate a plurality of internal parameters and a plurality of deformation parameters.

[0009] In some embodiments, multiple internal parameters are coordinate system transformations between the camera coordinate system and a two-dimensional pixel coordinate system corresponding to one of the N images. These multiple internal parameters are affected by a focal length, an optical center, and a skew coefficient of the camera. These multiple internal parameters are stored and used to adjust an image frame of the camera when the camera captures another image that does not involve the calibration chart.

[0010] In some embodiments, the plurality of deformation parameters are multiple nonlinear lens deformations of the camera. The plurality of deformation parameters are stored and used to adjust an image frame of the camera when the camera captures another image that does not involve the calibration chart.

[0011] In some embodiments, the control method further includes: capturing N images involving the calibration chart with the camera, where N is a positive integer greater than 1; generating N third rotation transformation matrices between the camera coordinate system and the calibration chart based on the calibration chart appearing in the N images; calculating N candidate rotation transformation matrices between the camera coordinate system and the first tracking device based on the first rotation transformation matrix, the second rotation transformation matrix, and the N third rotation transformation matrices; statistically analyzing the N candidate rotation transformation matrices; and calculating the fourth rotation transformation matrix based on the analysis results of the N candidate rotation transformation matrices.

[0012] Another embodiment of this disclosure discloses a tracking system including a camera, a first trackable device, a second trackable device, a tracking base station, and a processing unit. The camera is used to capture at least one image relating to a calibration chart. The first trackable device is attached to the camera. The second trackable device is attached to the calibration chart. The tracking base station is used to track the first and second trackable devices to generate a first rotational transformation matrix between the first trackable device and the tracking base station, and to generate a second rotational transformation matrix between the second trackable device and the tracking base station. The processing unit is communicatively connected to the tracking base station and the camera, wherein the processing unit is used to generate a third rotational transformation matrix between a camera coordinate system and the calibration chart based on the calibration chart appearing in the at least one image; the processing unit is used to calculate a fourth rotational transformation matrix between the camera coordinate system and the first trackable device based on the first, second, and third rotational transformation matrices; and the processing unit is used to track the camera based on the first and fourth rotational transformation matrices.

[0013] In some embodiments, the calibration chart includes a feature pattern and a carrier socket, the second trackable device entity is attached to the carrier socket and has a mechanical configuration relationship relative to the feature pattern, a fifth rotational transformation matrix between the calibration chart and the second trackable device entity is obtained from the mechanical configuration relationship, and the calculation of the fourth rotational transformation matrix is ​​further based on the fifth rotational transformation matrix.

[0014] In some embodiments, the fourth rotation transformation matrix is ​​calculated as the product of the third rotation transformation matrix, the fifth rotation transformation matrix, the second rotation transformation matrix, and the first rotation transformation matrix.

[0015] In some embodiments, the origin of the camera coordinate system is located at an optical center of the camera, and the fourth rotation transformation matrix is ​​used to describe a rotational relationship and a positional relationship between the camera coordinate system and the first tracking device.

[0016] In some embodiments, the processing unit is used to perform a camera geometry calibration of the camera based on the calibration chart in the at least one image to generate a plurality of internal parameters and a plurality of deformation parameters.

[0017] In some embodiments, the plurality of internal parameters are coordinate system transformations between the camera coordinate system and a two-dimensional pixel coordinate system corresponding to the at least one image. The plurality of internal parameters are affected by a focal length, an optical center, and a skew coefficient of the camera. The plurality of internal parameters are stored and used to adjust an image frame of the camera when the camera captures another image that does not involve the calibration chart.

[0018] In some embodiments, the plurality of deformation parameters are multiple nonlinear lens deformations of the camera. The plurality of deformation parameters are stored and used to adjust an image frame of the camera when the camera captures another image that does not involve the calibration chart.

[0019] In some embodiments, the camera captures N images involving the calibration chart, where N is a positive integer greater than 1. Based on the calibration chart appearing in the N images, the processing unit generates N third rotational transformation matrices between the camera coordinate system and the calibration chart. Based on the first rotational transformation matrix, the second rotational transformation matrix, and the N third rotational transformation matrices, the processing unit calculates N candidate rotational transformation matrices between the camera coordinate system and the first tracking device. The processing unit statistically analyzes the N candidate rotational transformation matrices. Based on the analysis results of the N candidate rotational transformation matrices, the processing unit calculates the fourth rotational transformation matrix.

[0020] Another embodiment of this disclosure discloses a non-transitory computer-readable medium storing at least one program instruction executable by a processing unit to run a tracking method. The tracking method includes the following steps: capturing at least one image involving a calibration chart with a camera, wherein a first trackable device entity is attached to the camera and a second trackable device entity is attached to the calibration chart; tracking the first trackable device and the second trackable device with a tracking base station to generate a first rotational transformation matrix between the first trackable device and the tracking base station, and generating a second rotational transformation matrix between the second trackable device and the tracking base station; generating a third rotational transformation matrix between a camera coordinate system and the calibration chart based on the calibration chart appearing in the at least one image; and calculating a fourth rotational transformation matrix between the camera coordinate system and the first trackable device based on the first, second, and third rotational transformation matrices, wherein the fourth rotational transformation matrix is ​​used to track the camera.

[0021] In some embodiments, the plurality of internal parameters relate to the coordinate system transformation between the camera coordinate system and the one-dimensional pixel coordinate system corresponding to one of the N images. The plurality of internal parameters are affected by a focal length, an optical center, and a skew coefficient of the camera. The plurality of internal parameters are stored. When the camera captures another image that does not involve the calibration chart, the stored plurality of internal parameters are used to adjust an image frame of the camera. The plurality of deformation parameters relate to a plurality of nonlinear lens deformations of the camera. The plurality of deformation parameters are stored. When the camera captures another image that does not involve the calibration chart, the stored plurality of deformation parameters are used to adjust an image frame of the camera.

[0022] In some embodiments, the tracking method includes: capturing N images involving the calibration chart with the camera, where N is a positive integer greater than 1; generating N third rotational transformation matrices between the camera coordinate system and the calibration chart based on the calibration chart appearing in the N images; calculating N candidate rotational transformation matrices between the camera coordinate system and the first trackable device based on the first rotational transformation matrix, the second rotational transformation matrix, and the N third rotational transformation matrices; statistically analyzing the N candidate rotational transformation matrices; and calculating the fourth rotational transformation matrix based on the analysis results of the N candidate rotational transformation matrices.

[0023] In this way, the aforementioned tracking system and control method can accurately track and locate the camera, making the objects captured by the camera appear more realistic when combined with the virtual scene.

[0024] It should be noted that the above description and the following detailed description are illustrative of this case by way of embodiments, and are used to assist in the explanation and understanding of the invention content claimed in this case. Attached Figure Description

[0025] To make the above and other objects, features and embodiments of this disclosure more apparent and understandable, the accompanying drawings are described below:

[0026] Figure 1 A schematic diagram of a tracking system according to some embodiments of the present disclosure is shown;

[0027] Figure 2 A schematic diagram of a tracking system having the function of tracking the optical center of a camera during the calibration process, according to some embodiments of this disclosure;

[0028] Figure 3 Show Figure 2 A flowchart illustrating the control method implemented by the tracking system during the calibration process;

[0029] Figure 4 The diagram illustrates the tracking system during the calibration process in some embodiments of this disclosure;

[0030] Figure 5 It shows Figure 2 A flowchart of the control method executed by the tracking system during the calibration process in another embodiment;

[0031] Figure 6A as well as Figure 6B A schematic diagram showing images captured by a camera at different positions relative to a calibration chart during a calibration process, according to an embodiment of this disclosure;

[0032] Figure 7A as well as Figure 7B A schematic diagram illustrating the tracking system performing the tracking function of the camera's optical center in a typical application after the calibration process is completed; and

[0033] Figure 8 This shows that after the calibration process is completed, by Figure 7A or Figure 7B The flowchart shows the control method performed by the tracking system.

[0034] Symbol explanation:

[0035] 100: Tracking System

[0036] 120: Camera

[0037] 141: First trackable device

[0038] 142: Second Trackable Device

[0039] 160: Tracking base stations

[0040] 180: Calibration Chart

[0041] 182: Feature Pattern

[0042] 184: Carrier Socket

[0043] 190: Processing Unit

[0044] 300a, 300b, 300c: Control methods

[0045] S310~S354: Steps

[0046] C141, C142, C160, C182: Reference Center

[0047] O141, O142, O160, O182: Coordinate system

[0048] C120: Optical Center

[0049] O120: Camera coordinate system

[0050] DIS: Deviation Distance

[0051] SA: Spatial Scope

[0052] OBJ: Object

[0053] RT1, RT1a, RT1n: First rotation transformation matrix

[0054] RT1x: The current first rotation transformation matrix

[0055] RT1y: The current first rotation transformation matrix

[0056] RT2, RT2a, RT2n: Second rotation transformation matrices

[0057] RT3, RT3a, RT3n: Third rotation transformation matrix

[0058] RT4: Fourth rotation transformation matrix

[0059] RT4a, RT4n: Candidate rotation transformation matrices

[0060] RT5: Fifth rotation transformation matrix

[0061] IMG, IMGa, IMGn: Images Detailed Implementation

[0062] The following disclosure provides numerous different embodiments or examples for implementing various features of this disclosure. Elements and configurations in the specific examples are used in the following discussion to simplify this disclosure. Any examples discussed are for illustrative purposes only and do not in any way limit the scope or meaning of this disclosure or its examples. Where appropriate, the same reference numerals are used between the drawings and in the corresponding text description to represent the same or similar elements.

[0063] Please see Figure 1 This illustrates a schematic diagram of a tracking system 100 according to some embodiments of the present disclosure. For example... Figure 1 As shown, the tracking system 100 includes a camera 120, a first trackable device 141, and a tracking base station 160 located within a spatial range SA. For example, as Figure 1 The spatial area SA shown may be a real-world film set or conference room, but this disclosure is not limited thereto. In some other embodiments, the spatial area SA may also be a specific area in an outdoor space (not shown in the figure).

[0064] In some embodiments, camera 120 can capture or photograph videos or images of real-world objects (OBJs). For example, the so-called real-world object OBJ could be a martial arts actor, and camera 120 could capture videos of the martial arts actor and integrate the real-world object OBJ into an immersive scene as one of its characters. The immersive scene may also additionally include virtual backgrounds (e.g., outer space) and other virtual objects (e.g., spaceships and aliens). To ensure the immersive scene looks more realistic, accurately tracking the position and orientation of camera 120 is crucial, as this allows for precise positioning of the video perspective captured by camera 120.

[0065] In some embodiments, the tracking system 100 includes a first trackable device 141 and a tracking base station 160. For example... Figure 1 As shown, the first trackable device 141 is physically attached to the camera 120. The tracking base station 160 is positioned at a fixed location within the spatial range SA. For example, the tracking base station 160 is positioned... Figure 1 The image shows a corner location near the ceiling of the room. As camera 120 moves, the first trackable device 141 moves accordingly. Tracking base station 160 can track the first trackable device 141, allowing it to determine the approximate location of camera 120 (based on the tracked location of the first trackable device 141). However, in this case, tracking base station 160 still cannot pinpoint the exact location of camera 120 because there is still a certain distance difference between the first trackable device 141 and camera 120.

[0066] In some embodiments, the tracking base station 160 may transmit optical tracking signals, and the first trackable device 141 may include an optical sensor (not shown) to sense the optical tracking signals emitted by the tracking base station 160, thereby tracking the spatial relationship between the first trackable device 141 and the tracking base station 160. However, this disclosure does not use the aforementioned optical sensing method to track the first trackable device 141. In some other embodiments, computer vision may also be used to track feature patterns provided on the first trackable device 141.

[0067] Based on the functions of the tracking base station 160 and the first trackable device 141, the tracking system 100 is able to track the reference center C141 of the first trackable device 141. Since the first trackable device 141 is physically attached to the camera 120, in some examples, the reference center C141 is directly taken as the position of the camera 120. However, as... Figure 1 As shown, there is still a certain deviation distance DIS between the reference center C141 of the first tracking device 141 and the optical center C120 of the camera 120.

[0068] If the reference center C141 of the first tracking device 141 is assumed to be the origin of the viewing angle of the camera 120, then the video captured by the camera 120 will be incorrectly assumed to be captured at an incorrect origin of viewing angle (i.e., reference center C141), and will deviate slightly from the true origin of viewing angle (i.e., optical center C120). Therefore, in order to correct the above deviation, it is desirable to know the deviation distance DIS between the reference center C141 and the optical center C120. The optical center C120 of the camera 120 is located inside the camera 120 and is affected by the lens, pixel sensor, and optical components of the camera 120 itself, making it very difficult to accurately determine the precise position of the optical center C120 of the camera 120 from its appearance. Therefore, the deviation distance DIS between the reference center C141 and the optical center C120 of the camera 120 is difficult to measure directly.

[0069] In some embodiments, the tracking system 100 provides a method for measuring the deviation distance DIS, thereby accurately tracking and calibrating the camera 120. Please refer to further details. Figure 2 The diagram illustrates a tracking system 100 that, according to some embodiments of this disclosure, has the function of tracking the optical center C120 of the tracking camera 120 during the calibration process.

[0070] In some embodiments, the processing unit 190 is communicatively connected to the camera 120, the first tracking device 141, and the second tracking device 142. The processing unit 190 may be a central processing unit, a graphics processing unit, a processor, and / or an application-specific integrated circuit (ASIC). In some embodiments, the processing unit 190 may be implemented by a stand-alone computer or a stand-alone server. In other embodiments, the processing unit 190 may be integrated into the camera 120 or the tracking base station 160.

[0071] In some embodiments, the tracking base station 160 may transmit optical tracking signals, and the second trackable device 142 may include an optical sensor (not shown) to sense the optical tracking signals emitted by the tracking base station 160, thereby tracking the spatial relationship between the second trackable device 142 and the tracking base station 160. However, this disclosure does not use the aforementioned optical sensing method to track the second trackable device 142. In some other embodiments, computer vision may also be used to track feature patterns provided on the second trackable device 142.

[0072] The calibration chart 180 includes a feature pattern 182 and a load socket 184. For example... Figure 2As shown, in some embodiments, feature pattern 182 can be a checkerboard pattern, comprising a plurality of white squares of a predetermined size and a plurality of black squares arranged at predetermined intervals. The checkerboard pattern facilitates geometric camera calibration of camera 120. For example, camera geometric calibration can be performed based on a pinhole camera model calibration method.

[0073] The second trackable device 142 is attached to the passenger socket 184 and its position has a certain mechanical configuration relationship with respect to the feature pattern. For example... Figure 2 As shown, when the second trackable device 142 is attached to the carrier socket 184, the reference center C142 of the second trackable device 142 is held in a fixed position by the limitation of the carrier socket 184. In this example, the reference center C142 of the second trackable device 142 will be fixed with a predetermined relative relationship to the reference center C182 of the feature pattern 182 on the calibration chart 180. This mechanical configuration relationship can be directly measured and manually set. For example, by forming the carrier socket 184 in the position desired by the designer, the reference center C142 can be precisely fixed at a position 30 cm to the left and 10 cm below the reference center C182 of the feature pattern 182. The spatial relationship between the second trackable device 142 and the feature pattern 182 can be described by a fifth rotation-translation matrix.

[0074] According to some embodiments, during the calibration process, camera 120 is triggered to capture at least one image IMG involving calibration chart 180. Based on the image IMG and the tracking results of the first tracking device 141 and the second tracking device 142, processing unit 190 is able to calculate the deviation distance between the reference center C141 of the first tracking device 141 and the optical center C120 of camera 120.

[0075] Please refer to the following: Figure 3 as well as Figure 4 , Figure 3 Show Figure 2 A flowchart of the control method 300a executed by the tracking system 100 during the calibration process. Figure 4 The diagram illustrates a tracking system 100 during the calibration process in some embodiments of this disclosure.

[0076] like Figure 2 , Figure 3 as well as Figure 4As shown, step S310 is executed to trigger camera 120, thereby causing camera 120 to capture an image IMG (such as an image from calibration chart 180) involving the calibration chart 180. Figure 2 (As shown).

[0077] Step S312 is executed by tracking the attitude data of the first trackable device 141 through the tracking base station 160, and then generating a first rotation transformation matrix RT1 between the first trackable device 141 and the tracking base station 160 based on the attitude data of the first trackable device 141. The first rotation transformation matrix RT1 describes the relative rotational and positional relationships between the coordinate system O141 of the first trackable device 141 and the coordinate system O160 of the tracking base station 160. In some embodiments, the first rotation transformation matrix RT1 is defined as a rotation transformation matrix. It is used to convert the original vector located in coordinate system O141 into an equivalent vector located in coordinate system O160.

[0078] Step S314 is executed by tracking the attitude data of the second trackable device 142 through the tracking base station 160, and then generating a second rotation transformation matrix RT2 between the second trackable device 142 and the tracking base station 160 based on the attitude data of the second trackable device 142. The second rotation transformation matrix RT2 describes the relative rotational and positional relationships between the coordinate system O142 of the second trackable device 142 and the coordinate system O160 of the tracking base station 160. In some embodiments, the second rotation transformation matrix RT2 is defined as a rotation transformation matrix. It is used to convert the original vector located in coordinate system O142 into an equivalent vector located in coordinate system O160.

[0079] Step S316 is performed to provide a fifth rotational transformation matrix RT5 between the calibration chart 180 and the second traceable device 142. In some embodiments, the fifth rotational transformation matrix RT5 is derived based on the spatial relationship between the second traceable device 142 and the feature pattern 182. Since the second traceable device 142 is mounted in a carrier socket 184 on the calibration chart 180, the position of the carrier socket 184 can be deliberately designed to give the fifth rotational transformation matrix RT5 a specific, predetermined value. The value of the fifth rotational transformation matrix RT5 is directly measurable and can be manually set (by adjusting the position of the carrier socket 184). In some embodiments, the fifth rotational transformation matrix RT5 is defined as... It is used to convert the original vector located in coordinate system O142 into an equivalent vector located in coordinate system O182.

[0080] like Figure 3In the embodiment shown, step S320 is performed to generate a third rotation transformation matrix RT3 between the camera coordinate system O120 of the camera 120 and the coordinate system O182 of the feature pattern 182 on the calibration chart 180, based on the calibration chart 180 appearing in the image IMG.

[0081] In some embodiments, in step S320, processing unit 190 uses a computer vision algorithm to generate a third rotation transformation matrix RT3 between the camera coordinate system O120 of camera 120 and the coordinate system O182 of feature pattern 182 based on the calibration chart 180 appearing in the image IMG. In some embodiments, the computer vision algorithm in step S320 can be used to sense the relative movement between camera 120 and calibration chart 180. For example, when feature pattern 182 appears smaller in the image IMG, the third rotation transformation matrix RT3 generated by processing unit 190 will indicate that camera coordinate system O120 is located further away from coordinate system O182. In another example, when feature pattern 182 appears larger in the image IMG, the third rotation transformation matrix RT3 generated by processing unit 190 will indicate that camera coordinate system O120 is located closer to coordinate system O182. In another example, when one side of feature pattern 182 appears larger than the opposite side in the image IMG, the third rotation transformation matrix RT3 generated by processing unit 190 will represent the rotation angle between camera coordinate system O120 and coordinate system O182. In some embodiments, the third rotation transformation matrix RT3 is defined as... It is used to convert the original vector located in coordinate system O182 into an equivalent vector located in camera coordinate system O120.

[0082] It should be noted that the origin of the camera coordinate system O120 is located at the optical center C120 inside the camera 120. The position of the optical center C120 is difficult to measure directly using mechanical methods because it is a theoretical location within the camera 120. In this example, the fourth rotation transformation matrix RT4 between the camera coordinate system O120 and the coordinate system O141 of the first tracking device 141 cannot be directly measured based on the physical connection between the first tracking device 141 and the camera 120. Figure 3 as well as Figure 4 As shown, step S320 is executed, and the processing unit 190 calculates the fourth rotation transformation matrix RT4 between the camera coordinate system O120 and the coordinate system O141 of the first tracking device 141 based on the aforementioned multiple rotation transformation matrices RT1, RT2, RT3 and RT5.

[0083] In some embodiments, the fourth rotation transformation matrix RT4 is defined as It is used to transform the original vector located in the camera coordinate system O120 into an equivalent vector located in the coordinate system O141 (first tracking device 141). The fourth rotation transformation matrix RT4 can be calculated in the following way:

[0084]

[0085]

[0086]

[0087] As described above, the processing unit 190 can calculate the fourth rotation transformation matrix RT4 based on the product of the third rotation transformation matrix RT3, the fifth rotation transformation matrix RT5, the second rotation transformation matrix RT2, and the first rotation transformation matrix RT1.

[0088] In some embodiments, the tracking system 100 and tracking method 300a may calculate a fourth rotation transformation matrix RT4 during the calibration process. The fourth rotation transformation matrix RT4 describes the relative rotational and positional relationships between the camera coordinate system O120 and the first trackable device 141. Since the first trackable device 141 is physically attached to the camera 120 at a fixed position, the fourth rotation transformation matrix RT4 calculated in step S330 maintains stable data. The origin of the camera coordinate system O120 is located at the optical center C120 of the camera 120.

[0089] In this way, by tracking the first trackable device 141 (and its coordinate system O141) and applying the fourth rotation transformation matrix RT4 to the tracking result of the first trackable device 141 (and its coordinate system O141), the tracking system 100 can track the accurate position of the optical center C120 of the camera 120 and the camera coordinate system O120.

[0090] like Figure 2 , Figure 3 as well as Figure 4 As shown, in step S340, processing unit 190 may store the fourth rotation transformation matrix RT4. In some embodiments, the fourth rotation transformation matrix RT4 may be stored in a digital data storage device (not shown), such as a memory, hard disk, cache memory, flash memory, or other similar data storage device. After the calibration procedure, the stored fourth rotation transformation matrix RT4 can be used at the optical center C120 of the tracking camera 120.

[0091] like Figure 3In the embodiment shown, control method 300a triggers camera 120 to capture an image IMG to calculate the fourth rotation transformation matrix RT4; however, this disclosure is not limited thereto.

[0092] In some embodiments, camera 120 requires some correction parameters to adjust the image frame of camera 120, such as multiple intrinsic parameters and / or multiple distortion parameters. In other words, based on the data of intrinsic parameters and / or distortion parameters, camera 120 can convert the sensing results of multiple pixel sensors into the frame of the image IMG. Please participate. Figure 5 It shows Figure 2 A flowchart of the control method 300b executed by the tracking system 100 during the calibration process in another embodiment is shown. Figure 5 In the illustrated embodiment, control method 300b can generate a fourth rotation transformation matrix RT4 (for tracking camera 120) and measure the camera's internal parameters and / or deformation parameters.

[0093] Figure 5 The control method 300b shown in the embodiment is similar to Figure 3 Control method 300a. One difference between control method 300b and control method 300a is that, for example... Figure 2 as well as Figure 5 As shown, in step S310, control method 300b triggers camera 120 to capture N images IMGa~IMGn involving calibration chart 180, where N is a positive integer greater than 1. For example, if N=5, camera 120 can capture five different images involving calibration chart 180. Control method 300b further includes step S315.

[0094] Please refer to the following: Figure 6A as well as Figure 6B It shows a schematic diagram of images IMGa~IMGn taken by camera 120 at different positions relative to calibration chart 180 during a calibration process according to an embodiment of this disclosure.

[0095] like Figure 6A As shown, camera 120 is positioned relative to calibration chart 180 at a specific location, and camera 120 captures an image IMGa; as Figure 6B As shown, camera 120 is located in a different position relative to calibration chart 180, and camera 120 captures another image IMGn. Figure 6A as well as Figure 6BAn example is shown where camera 120 takes two images, IMGa and IMGn, from two different locations. However, this disclosure is not limited to this; camera 120 can be moved to many more different locations to take more images.

[0096] In step S315, based on the feature patterns 182 appearing in the calibration charts 180 within the multiple images IMGa~IMGn, the processing unit 190 can perform geometric camera calibration on the camera 120 to generate intrinsic and deformation parameters. In some embodiments, camera geometric calibration is an estimation procedure based on a camera model to estimate the intrinsic and deformation parameters corresponding to the camera model, enabling the camera model to approximate the captured image (i.e., ...) Figure 2 The feature pattern 182) of the real camera 120 is used to calculate the intrinsic parameters and deformation parameters. It should be noted that this disclosure is not limited to the pinhole camera model; other camera models can also be used to generate intrinsic and deformation parameters. Detailed methods for performing camera geometric correction based on a camera model have been widely discussed in the relevant field and are well known to those skilled in the art, and will not be elaborated upon here.

[0097] The intrinsic parameters pertain to the transformation between two coordinate systems, specifically from the camera coordinate system O120 to the two-dimensional pixel coordinate system of the image IMG (corresponding to the multiple pixel sensors of camera 120, not shown in the figure). These intrinsic parameters are influenced by the internal configuration of camera 120, such as its focal length, optical center C120, and skew coefficient.

[0098] In this example, the multiple internal parameters calculated in step S315 can be stored. When camera 120 captures another image that does not involve calibration chart 180, these stored internal parameters can be used to adjust the image frame of camera 120. For example, after the calibration process is complete, when camera 120 is capturing the relevant object OBJ (see...) Figure 1 When shooting images, videos, or movies, the stored internal parameters can be used to adjust the image frame of the camera 120.

[0099] The multiple deformation parameters are multiple nonlinear lens deformations of the camera 120. In some embodiments, the processing unit 190 may calculate multiple deformation parameters at the same time when performing camera geometry correction.

[0100] In this example, the multiple deformation parameters calculated in step S315 can be stored. When the camera captures another image that does not involve the calibration chart 180, the stored deformation parameters are used to adjust the image frame of the camera 120. For example, after the calibration process is complete, when the camera 120 is capturing the relevant object OBJ (see... Figure 1 When shooting images, videos, or movies, the stored deformation parameters can be used to adjust the image frame of the camera 120.

[0101] It should be noted that, ideally, the internal parameters calculated for the N images IMGa~IMGn in step S315 will be the same, and the deformation parameters calculated for the N images IMGa~IMGn in step S315 will also be the same. This is because the internal parameters and deformation parameters are determined by the internal factors of the camera 120.

[0102] In step S320, control method 300b uses computer vision to generate N third rotation transformation matrices RT3a~RT3n between the camera coordinate system O120 of camera 120 and the calibration map 180 based on feature patterns 182 appearing on the calibration map 180 in N images IMGa~IMGn. The generation method of each of the N third rotation transformation matrices RT3a~RT3n is similar to that in the previous embodiment. Figure 3 The step S320 shown (generates a third rotation transformation matrix RT3 based on a single image).

[0103] In step S331, the processing unit 190 calculates N candidate rotation transformation matrices RT4a~RT4n between the camera coordinate system O120 and the first tracking device 141 based on N first rotation transformation matrices RT1a~RT1n, N second rotation transformation matrices RT2a~RT2n, N third rotation transformation matrices RT3a~RT3n, and a fifth rotation transformation matrix RT5.

[0104] related Figure 5 The detailed implementation of steps S310 to S331 of control method 300b can be achieved by executing steps S310 to S330 of control method 300a N times and moving the camera to different positions during the process.

[0105] like Figure 5 As shown, step S332 is executed, and the processing unit 190 analyzes the N candidate rotation transformation matrices RT4a~RT4n in a statistical manner, thereby calculating the fourth rotation transformation matrix RT4 based on the analysis results of the N candidate rotation transformation matrices RT4a~RT4n.

[0106] In some embodiments, a fourth rotation transformation matrix RT4 can be generated based on the average value of N candidate rotation transformation matrices RT4a~RT4n.

[0107] In some embodiments, a fourth rotation transformation matrix RT4 can be generated based on the intermediate values ​​of N candidate rotation transformation matrices RT4a~RT4n.

[0108] In some embodiments, the standard deviation can be calculated based on N candidate rotation transformation matrices RT4a~RT4n, and then the standard deviation can be used to determine whether each of the N candidate rotation transformation matrices RT4a~RT4n is reliable. Control method 300b can remove unreliable candidate matrices and calculate the fourth rotation transformation matrix RT4 based only on reliable candidate matrices.

[0109] Step S340 is executed to store the fourth rotation transformation matrix RT4, internal parameters, and deformation parameters.

[0110] Figure 5 The control method 300b shown can calculate the fourth rotation transformation matrix RT4 based on N different images IMGa~IMGn during the correction process, which helps to eliminate potential errors that may occur during camera geometry correction in step S320.

[0111] Please refer to the following: Figure 7A as well as Figure 7B . Figure 7A as well as Figure 7B This diagram illustrates the tracking function of the tracking system 100 in a general application after the calibration process is completed, tracking the optical center C120 of the camera 120. Figure 7A as well as Figure 7B As shown, in general applications (e.g., when filming a video of an object OBJ with camera 120), camera 120 may move to different positions within the spatial range SA.

[0112] like Figure 7A As shown, camera 120 moves to the left side of the spatial range SA. In this example, the tracking base station 160 of the tracking system 100 can sense the first trackable device 141 to generate a current first rotation transformation matrix RT1x. Based on the current first rotation transformation matrix RT1x and a pre-stored fourth rotation transformation matrix RT4, the tracking system 100 can accurately track camera 120.

[0113] like Figure 7BAs shown, camera 120 moves to the right side of the spatial range SA. In this example, the tracking base station 160 of the tracking system 100 can sense the first trackable device 141 to generate a current first rotation transformation matrix RT1y. Based on the current first rotation transformation matrix RT1y and a pre-stored fourth rotation transformation matrix RT4, the tracking system 100 can accurately track camera 120.

[0114] In this case, when the object OBJ captured by the camera 120 is merged with the virtual background (such as outer space) and virtual objects (such as spaceships and aliens) of the immersive scene, the perspective of the captured object OBJ can be accurately tracked and positioned. In this way, the object OBJ appearing in the immersive scene can look more realistic.

[0115] Please refer to the following: Figure 8 It shows that after the correction process is completed, by Figure 7A or Figure 7B The flowchart shows the control method 300c executed by the tracking system 100.

[0116] like Figure 7A , Figure 7B as well as Figure 8 As shown, step S351 is performed, whereby the tracking base station 160 tracks the first trackable device 141 to generate the current first rotation transformation matrix RT1x / RT1y. Step S352 is performed, whereby the camera 120 is tracked based on the current first rotation transformation matrix RT1x / RT1y and the fourth rotation transformation matrix RT4 stored in the previous calibration process. Step S353 is performed, whereby the camera 120 captures an image of the non-calibration chart (see [reference]). Figure 2 The image of the calibration chart 180 shown. For example, the camera 120 can capture images, streamed images, or videos related to different objects (such as actors, buildings, animals, or scenery). Step S354 is performed to adjust the image frame of the camera 120 according to the stored internal parameters and the stored deformation parameters.

[0117] Another embodiment of this disclosure is a non-transitory computer-readable medium that stores at least one program instruction that can be processed by a processing unit (see previous embodiments and...). Figure 2 The processing unit 190 shown executes the following: Figure 3 , Figure 5 and Figure 8 The tracking methods shown are 300a, 300b, and 300c.

[0118] While specific embodiments of this disclosure have been disclosed with respect to the above embodiments, these embodiments are not intended to limit this disclosure. Various alternatives and modifications can be made by those skilled in the art without departing from the principles and spirit of this disclosure. Therefore, the scope of protection of this disclosure is defined by the appended claims.

Claims

1. A control method, characterized in that, This control method includes: A camera captures at least one image relating to a calibration chart, wherein a first trackable device entity is attached to the camera and a second trackable device entity is attached to the calibration chart. A tracking base station tracks the first trackable device and the second trackable device to generate a first rotation transformation matrix between the first trackable device and the tracking base station, and to generate a second rotation transformation matrix between the second trackable device and the tracking base station; Based on the calibration chart appearing in the at least one image, a camera coordinate system of the camera and a third rotation transformation matrix between the calibration chart are generated; as well as Based on the first rotation transformation matrix, the second rotation transformation matrix, the third rotation transformation matrix, and a fifth rotation transformation matrix between the calibration chart and the second trackable device entity, a fourth rotation transformation matrix between the camera coordinate system and the first trackable device is calculated, wherein the fourth rotation transformation matrix is ​​used to track the camera.

2. The control method of claim 1, wherein the calibration chart includes a feature pattern and a carrier socket, the second trackable device entity is attached to the carrier socket and has a mechanical configuration relationship relative to the feature pattern, and the fifth rotational transformation matrix between the calibration chart and the second trackable device entity is derived from the mechanical configuration relationship.

3. The control method as described in claim 2, wherein the fourth rotation transformation matrix is ​​calculated based on the product of the third rotation transformation matrix, the fifth rotation transformation matrix, the second rotation transformation matrix, and the first rotation transformation matrix.

4. The control method as claimed in claim 1, wherein the origin of the camera coordinate system is located at an optical center of the camera, and the fourth rotation transformation matrix is ​​used to describe a rotational relationship and a positional relationship between the camera coordinate system and the first tracking device.

5. The control method as described in claim 1, further comprising: The camera captures N images related to the calibration chart, where N is a positive integer greater than 1; and Based on the calibration chart appearing in the N images, a camera geometry calibration is performed to generate multiple intrinsic parameters and multiple deformation parameters.

6. The control method as described in claim 5, wherein the plurality of internal parameters are coordinate system transformations between the camera coordinate system and a two-dimensional pixel coordinate system corresponding to one of the N images, the plurality of internal parameters are affected by a focal length, an optical center, and a skew coefficient of the camera, the plurality of internal parameters are stored, and when the camera captures another image not involving the calibration chart, the stored plurality of internal parameters are used to adjust an image frame of the camera.

7. The control method of claim 5, wherein the plurality of deformation parameters are a plurality of nonlinear lens deformations of the camera, the plurality of deformation parameters are stored, and when the camera captures another image not involving the calibration chart, the stored plurality of deformation parameters are used to adjust an image frame of the camera.

8. The control method as described in claim 1, further comprising: The camera captures N images related to the calibration chart, where N is a positive integer greater than 1; Based on the calibration chart appearing in the N images, generate the camera coordinate system of the camera and N third rotation transformation matrices between the calibration chart; Based on the first rotation transformation matrix, the second rotation transformation matrix, the N third rotation transformation matrices, and the fifth rotation transformation matrix, calculate N candidate rotation transformation matrices between the camera coordinate system and the first tracking device; Analyze the N candidate rotation transformation matrices using statistical methods; as well as Based on the analysis results of the N candidate rotation transformation matrices, the fourth rotation transformation matrix is ​​calculated.

9. A tracking system, characterized in that, The tracking system includes: A camera for capturing at least one image relating to a calibration chart; A first trackable device is physically attached to the camera; A second traceable device is physically attached to the calibration chart; A tracking base station is used to track the first trackable device and the second trackable device to generate a first rotation transformation matrix between the first trackable device and the tracking base station, and to generate a second rotation transformation matrix between the second trackable device and the tracking base station; as well as A processing unit, communicatively connected to the tracking base station and the camera, wherein the processing unit is used to: Based on the calibration chart appearing in the at least one image, a camera coordinate system of the camera and a third rotation transformation matrix between the calibration chart are generated; Based on the first rotation transformation matrix, the second rotation transformation matrix, the third rotation transformation matrix, and a fifth rotation transformation matrix between the calibration chart and the second trackable device entity, a fourth rotation transformation matrix between the camera coordinate system and the first trackable device is calculated; as well as The camera is tracked using the first tracking device and the fourth rotational transformation matrix.

10. The tracking system of claim 9, wherein the calibration chart includes a feature pattern and a carrier socket, the second trackable device entity is attached to the carrier socket and has a mechanical configuration relationship relative to the feature pattern, and the fifth rotational transformation matrix between the calibration chart and the second trackable device entity is obtained from the mechanical configuration relationship.

11. The tracking system of claim 10, wherein the fourth rotation transformation matrix is ​​calculated as the product of the third rotation transformation matrix, the fifth rotation transformation matrix, the second rotation transformation matrix, and the first rotation transformation matrix.

12. The tracking system of claim 9, wherein the origin of the camera coordinate system is located at an optical center of the camera, and the fourth rotation transformation matrix is ​​used to describe a rotational relationship and a positional relationship between the camera coordinate system and the first trackable device.

13. The tracking system of claim 9, wherein the processing unit is configured to perform a camera geometry calibration of the camera based on the calibration chart in the at least one image to generate a plurality of internal parameters and a plurality of deformation parameters.

14. The tracking system of claim 13, wherein the plurality of internal parameters are coordinate system transformations between the camera coordinate system and a two-dimensional pixel coordinate system corresponding to the at least one image, the plurality of internal parameters are affected by a focal length, an optical center, and a skew coefficient of the camera, the plurality of internal parameters are stored, and when the camera captures another image not involving the calibration chart, the stored plurality of internal parameters are used to adjust an image frame of the camera.

15. The tracking system of claim 13, wherein the plurality of deformation parameters are plurality of nonlinear lens deformations of the camera, the plurality of deformation parameters are stored, and when the camera captures another image not involving the calibration chart, the stored plurality of deformation parameters are used to adjust an image frame of the camera.

16. The tracking system of claim 9, wherein the camera captures N images relating to the calibration chart, where N is a positive integer greater than 1. Based on the calibration chart appearing in the N images, the processing unit generates N third rotation transformation matrices between the camera coordinate system and the calibration chart; Based on the first rotation transformation matrix, the second rotation transformation matrix, the N third rotation transformation matrices, and the fifth rotation transformation matrix, the processing unit calculates N candidate rotation transformation matrices between the camera coordinate system and the first tracking device; The processing unit analyzes the N candidate rotation transformation matrices in a statistical manner; as well as Based on the analysis results of the N candidate rotation transformation matrices, the processing unit calculates the fourth rotation transformation matrix.

17. A non-transitory computer-readable medium, characterized in that, The non-transitory computer-readable medium stores at least one program instruction which is executed by a processing unit to run a tracing method, the tracing method comprising: A camera captures at least one image relating to a calibration chart, wherein a first trackable device entity is attached to the camera and a second trackable device entity is attached to the calibration chart. A tracking base station tracks the first trackable device and the second trackable device to generate a first rotation transformation matrix between the first trackable device and the tracking base station, and to generate a second rotation transformation matrix between the second trackable device and the tracking base station; Based on the calibration chart appearing in the at least one image, a camera coordinate system of the camera and a third rotation transformation matrix between the calibration chart are generated; as well as Based on the first rotation transformation matrix, the second rotation transformation matrix, the third rotation transformation matrix, and a fifth rotation transformation matrix between the calibration chart and the second trackable device entity, a fourth rotation transformation matrix between the camera coordinate system and the first trackable device is calculated, wherein the fourth rotation transformation matrix is ​​used to track the camera.

18. The non-transitory computer-readable medium of claim 17, wherein the tracking method comprises: The camera captures N images related to the calibration chart, where N is a positive integer greater than 1; and Based on the calibration chart appearing in the N images, a camera geometry calibration is performed to generate multiple intrinsic parameters and multiple deformation parameters.

19. The non-transitory computer-readable medium of claim 18, wherein the plurality of internal parameters are coordinate system transformations between the camera coordinate system and a two-dimensional pixel coordinate system corresponding to one of the N images, the plurality of internal parameters are affected by a focal length, an optical center, and a skew coefficient of the camera, the plurality of internal parameters are stored, and when the camera captures another image not involving the calibration chart, the stored plurality of internal parameters are used to adjust an image frame of the camera, the plurality of deformation parameters are non-linear lens deformations of the camera, the plurality of deformation parameters are stored, and when the camera captures another image not involving the calibration chart, the stored plurality of deformation parameters are used to adjust an image frame of the camera.

20. The non-transitory computer-readable medium of claim 17, wherein the tracking method comprises: The camera captures N images related to the calibration chart, where N is a positive integer greater than 1; Based on the calibration chart appearing in the N images, generate the camera coordinate system of the camera and N third rotation transformation matrices between the calibration chart; Based on the first rotation transformation matrix, the second rotation transformation matrix, the N third rotation transformation matrices, and the fifth rotation transformation matrix, calculate N candidate rotation transformation matrices between the camera coordinate system and the first tracking device; Analyze the N candidate rotation transformation matrices using statistical methods; as well as Based on the analysis results of the N candidate rotation transformation matrices, the fourth rotation transformation matrix is ​​calculated.

Images