System and method for calibrating imaging and spatial orientation sensors

Abstract

Methods of calibrating the relative orientation between spatial orientation sensors, the relative orientation between imaging sensors and spatial orientation sensors, and the relative orientation between imaging sensors are disclosed. The relative orientation of the spatial orientation sensor and the imaging sensor is calculated relative to the reference coordinate system.

Description

[0001] This invention patent application is a divisional application of the Chinese invention patent application with application number 201880059859.6, PCT application number PCT / US2018 / 049469, PCT application date September 5, 2018, and invention name “System and method for calibrating imaging and spatial orientation sensors”.

[0002] Cross-reference to related applications

[0003] The disclosure of U.S. Provisional Application No. 62 / 557,786 is incorporated herein by reference in its entirety. Technical Field

[0004] The present disclosure relates to relative orientation calibration between spatial orientation sensors, relative orientation calibration between imaging sensors and spatial orientation sensors, and relative orientation calibration between imaging sensors and imaging sensors. The system can work in a calibration mode or a tracking mode. Background Art

[0005] Human-computer interaction (HCI), or more generally, human-computer interaction, focuses on computer technology and the design and use of interfaces between users and computers. HCI relies on sensitive, intuitive, and accurate measurement of human input actions. The mouse, keyboard, and touch screen are traditional input devices that require manual control by the user. Some input devices, such as Microsoft The user's body or hand gestures can be tracked without any physical contact. In this disclosure, the term "user" and the term "person" can be used interchangeably.

[0006] Recent advances in virtual reality (VR) technology have brought VR goggles to the consumer market. VR goggles can create an immersive three-dimensional (3D) experience for users. Users can look around in the virtual world by rotating their heads, just as they would look around in the real world.

[0007] Augmented reality (AR) is another area that is growing rapidly. A major difference between AR and VR is that AR operates in real-world scenes in real time, as opposed to VR, which operates only in scenes created or recorded by computers. In both VR and AR, it is very useful to know where the user is looking and what actions the user wants to take with respect to the desired target. Effective and reliable eye tracking will enable widespread application in this context.

[0008] Autonomous vehicles are also on the cutting edge today. There are situations where a car in autonomous mode may require the driver's attention due to updates on road or traffic conditions or changes in driving patterns. Therefore, it is useful to continuously monitor where the driver is looking.

[0009] Machine learning and artificial intelligence (AI) can work in a cycle of learning, modeling, and prediction. In this cycle, fast and intuitive tracking of user attention for data collection and confirmation can play an important role. Summary of the invention

[0010] This article discloses a method, including: obtaining a first orientation of a first spatial orientation sensor and a first orientation of a second spatial orientation sensor; obtaining a second orientation of the first spatial orientation sensor and a second orientation of the second spatial orientation sensor; obtaining a third orientation of the first spatial orientation sensor and a third orientation of the second spatial orientation sensor; obtaining a fourth orientation of the first spatial orientation sensor and a fourth orientation of the second spatial orientation sensor; determining a relative orientation of the first spatial orientation sensor relative to the second spatial orientation sensor based on the first orientation of the first spatial orientation sensor, the second orientation of the first spatial orientation sensor, the third orientation of the first spatial orientation sensor, the fourth orientation of the first spatial orientation sensor, the first orientation of the second spatial orientation sensor, the second orientation of the second spatial orientation sensor, the third orientation of the second spatial orientation sensor and the fourth orientation of the second spatial orientation sensor; wherein the relative orientation of the first spatial orientation sensor relative to the second spatial orientation sensor is fixed.

[0011] According to an embodiment, at least three of the first orientation of the first spatial orientation sensor, the second orientation of the first spatial orientation sensor, the third orientation of the first spatial orientation sensor, and the fourth orientation of the first spatial orientation sensor are different from each other; wherein the first orientation of the first spatial orientation sensor is different from the second orientation of the first spatial orientation sensor; wherein the third orientation of the first spatial orientation sensor is different from the fourth orientation of the first spatial orientation sensor.

[0012] According to an embodiment, at least three of the first orientation of the second spatial orientation sensor, the second orientation of the second spatial orientation sensor, the third orientation of the second spatial orientation sensor, and the fourth orientation of the second spatial orientation sensor are different from each other; wherein the first orientation of the second spatial orientation sensor is different from the second orientation of the second spatial orientation sensor; wherein the third orientation of the second spatial orientation sensor is different from the fourth orientation of the second spatial orientation sensor.

[0013] According to an embodiment, determining the relative orientation of a first spatial orientation sensor relative to a second spatial orientation sensor includes: determining the relative orientation of the first spatial orientation sensor relative to a reference system; determining the relative orientation of the second spatial orientation sensor relative to the reference system; and determining the relative orientation of the first spatial orientation sensor relative to the second spatial orientation sensor based on the relative orientation of the first spatial orientation sensor relative to the reference system and the relative orientation of the second spatial orientation sensor relative to the reference system.

[0014] According to an embodiment, the relative orientation of the first spatial orientation sensor relative to the reference system is determined based on the first orientation of the first spatial orientation sensor, the second orientation of the first spatial orientation sensor, the third orientation of the first spatial orientation sensor, and the fourth orientation of the first spatial orientation sensor.

[0015] According to an embodiment, the relative position of the second spatial position sensor relative to the reference system is determined based on the first position of the second spatial position sensor, the second position of the second spatial position sensor, the third position of the second spatial position sensor, and the fourth position of the second spatial position sensor.

[0016] This article discloses a method, comprising: using an imaging sensor to acquire an image of a first point in a three-dimensional space and an image of a second point in the three-dimensional space; based on the image of the first point and the image of the second point, determining the relative orientation of the imaging sensor with respect to a reference system; wherein the first point and the second point are fixed in the three-dimensional space.

[0017] According to an embodiment, the step of determining the relative orientation of the imaging sensor relative to the reference system further includes: determining a vector between a reference point fixed relative to the imaging sensor and the image of the first point, and determining a vector between the reference point and the image of the second point; and determining the relative orientation based on the vector and the vector.

[0018] According to an embodiment, the vector between the reference point and the image of the first point is a unit vector; wherein the vector between the reference point and the image of the second point is a unit vector.

[0019] This article discloses a method, including: using an imaging sensor at its first orientation to acquire a first image of a first point in a three-dimensional space and a second image of a second point in the three-dimensional space; using the imaging sensor at its second orientation to acquire a third image of the first point and a fourth image of the second point; based on the first image, the second image, the third image and the fourth image, determining the relative rotation of the imaging sensor between the first orientation and the second orientation; wherein the first point and the second point are fixed in the three-dimensional space.

[0020] According to an embodiment, the step of determining the relative rotation also includes: determining the relative orientation of the imaging sensor relative to the reference system when it is in the first orientation based on the first image and the second image; determining the relative orientation of the imaging sensor relative to the reference system when it is in the second orientation based on the third image and the fourth image; and determining the relative rotation of the imaging sensor between the first orientation and the second orientation based on the relative orientation of the imaging sensor relative to the reference system when it is in the first orientation and the relative orientation of the imaging sensor relative to the reference system when it is in the second orientation.

[0021] According to an embodiment, determining the relative orientation of the imaging sensor relative to the reference system when in a first orientation includes: determining a vector between a reference point fixed relative to the imaging sensor and an image of the first point; and determining a vector between the reference point and the image of the second point.

[0022] A method is disclosed herein, comprising: determining a first relative rotation of an imaging sensor between a first orientation of the imaging sensor and a second orientation of the imaging sensor; determining a second relative rotation of the imaging sensor between a third orientation of the imaging sensor and a fourth orientation of the imaging sensor; and determining a relative orientation of the imaging sensor relative to a reference system based on the first relative rotation and the second relative rotation.

[0023] According to an embodiment, a method for determining a first relative rotation includes: using an imaging sensor at its first orientation to acquire a first image of a first point in a three-dimensional space and a second image of a second point in the three-dimensional space; using an imaging sensor at its second orientation to acquire a third image of the first point and a fourth image of the second point; and determining the first relative rotation based on the first image, the second image, the third image, and the fourth image.

[0024] According to an embodiment, at least three of the first orientation, the second orientation, the third orientation, and the fourth orientation are different from each other; wherein the first orientation is different from the second orientation, and the third orientation is different from the fourth orientation.

[0025] The present invention discloses a method, comprising: determining the relative orientation of an imaging sensor relative to a reference system based on a first relative rotation of the imaging sensor between a first orientation of the imaging sensor and a second orientation of the imaging sensor and a second relative rotation of the imaging sensor between a third orientation of the imaging sensor and a fourth orientation of the imaging sensor; determining a first output, a second output, a third output and a fourth output corresponding to a spatial orientation sensor when the imaging sensor is in the first orientation, the second orientation, the third orientation and the fourth orientation, respectively; determining the relative orientation of the spatial orientation sensor relative to the reference system based on the first output, the second output, the third output and the fourth output; determining the relative orientation of the spatial orientation sensor relative to the imaging sensor based on the relative orientation of the spatial orientation sensor relative to the reference system and the relative orientation of the imaging sensor relative to the reference system.

[0026] According to an embodiment, the relative position of the spatial position sensor with respect to the imaging sensor is fixed.

[0027] According to an embodiment, the method also includes: determining a first relative rotation: using an imaging sensor at its first orientation to acquire a first image of a first point in three-dimensional space and a second image of a second point in three-dimensional space; using an imaging sensor at its second orientation to acquire a third image of the first point and a fourth image of the second point; based on the first image, the second image, the third image and the fourth image, determining the first relative rotation.

[0028] According to an embodiment, at least three of the first orientation, the second orientation, the third orientation and the fourth orientation are different from each other; wherein the first orientation is different from the second orientation, and the third orientation is different from the fourth orientation.

[0029] The present invention discloses a method, comprising: determining a first relative orientation of a first spatial orientation sensor relative to a first imaging sensor based on a relative orientation of the first spatial orientation sensor relative to a first reference system and a relative orientation of the first imaging sensor relative to the first reference system; determining a second relative orientation of the second spatial orientation sensor relative to the second imaging sensor based on a relative orientation of the second spatial orientation sensor relative to the second reference system and a relative orientation of the second imaging sensor relative to the second reference system; determining a third relative orientation of the first spatial orientation sensor relative to the second spatial orientation sensor; and determining a fourth relative orientation of the first imaging sensor relative to the second imaging sensor based on the first relative orientation, the second relative orientation and the third relative orientation.

[0030] According to an embodiment, the first spatial position sensor, the second spatial position sensor, the first imaging sensor and the second imaging sensor have a fixed relative position with respect to each other.

[0031] This article discloses a method, including: using an imaging sensor at its first orientation to acquire a first image of a first point in a three-dimensional space and a second image of a second point in the three-dimensional space; using the imaging sensor at its second orientation to acquire a third image of the first point and a fourth image of the second point; acquiring a first output and a second output corresponding to the spatial orientation sensor when the imaging sensor is in the first orientation and the second orientation respectively; and determining the focal length of the imaging sensor based on the first image, the second image, the third image, the fourth image, the first output and the second output.

[0032] According to an embodiment, a method of determining a focal length includes determining a relative rotation of an imaging sensor between a first orientation and a second orientation based on a first image, a second image, a third image, and a fourth image.

[0033] According to an embodiment, the imaging sensor and the spatial orientation sensor have a fixed relative orientation with respect to each other.

[0034] This article discloses a method, including: obtaining a first output, a second output, a third output, and a fourth output corresponding to a spatial orientation sensor when the spatial orientation sensor is in a first orientation, a second orientation, a third orientation, and a fourth orientation, respectively; and determining the relative orientation of the spatial orientation sensor relative to a reference system based on the first output, the second output, the third output, and the fourth output.

[0035] According to an embodiment, at least three of the first orientation, the second orientation, the third orientation and the fourth orientation are different from each other; wherein the first orientation is different from the second orientation; and the third orientation is different from the fourth orientation.

[0036] A method is disclosed herein, comprising: obtaining a first relative position of a person's eye relative to the first imaging sensor based on an image obtained using the first imaging sensor; obtaining a second relative position of a second imaging sensor relative to the first imaging sensor; and determining a third relative position of the eye relative to the second imaging sensor based on the first relative position and the second relative position.

[0037] Disclosed herein is a non-transitory machine-readable medium having instructions recorded thereon. When the instructions are executed by a computer, the instructions implement any of the above methods.

[0038] The present invention discloses an apparatus comprising: a first imaging sensor; a second imaging sensor; a first spatial orientation sensor; a second spatial orientation sensor; wherein the first spatial orientation sensor, the second spatial orientation sensor, the first imaging sensor, and the second imaging sensor have a fixed relative orientation relative to each other; and a computing device comprising a processor and the above-mentioned non-temporary machine-readable medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A two-dimensional (2D) coordinate system is shown.

[0040] Figure 2 An image of a pair of points p_a_i and p_b_i whose positions are fixed in the world coordinate system CS-W is shown.

[0041] Figure 3 A three-dimensional coordinate system is shown.

[0042] Figure 4 Several three-dimensional coordinate systems mentioned here are shown.

[0043] Figure 5 It is shown that a spatial orientation sensor can be attached to the camera.

[0044] Figure 6 The eye camera and field of view camera are shown attached to a harness frame worn by the user.

[0045] Fig. 7A A flow chart of a method according to an embodiment is schematically shown.

[0046] Figure 7B Shows Fig. 7A A flow chart of the steps in a flowchart.

[0047] Fig. 8AA flow chart of a method according to an embodiment is schematically shown.

[0048] Figure 8B Shows Fig. 8A A flow chart of the steps in a flowchart.

[0049] Fig. 9A A flow chart of a method according to an embodiment is schematically shown.

[0050] Fig. 9B Shows Fig. 9A A flow chart of the steps in a flowchart.

[0051] Fig. 10A A flow chart of a method according to an embodiment is schematically shown.

[0052] Fig. 10B Shows Fig. 10A A flow chart of the steps in a flowchart.

[0053] Fig.11A A flow chart of a method according to an embodiment is schematically shown.

[0054] Fig. 11B Shows Fig.11A A flow chart of the steps in a flowchart.

[0055] Fig.12 A flow chart of a method according to an embodiment is schematically shown.

[0056] Fig.13 A flow chart of a method according to an embodiment is schematically shown.

[0057] Fig.14 A flow chart of a method according to an embodiment is schematically shown.

[0058] Fig.15 A flow chart of a method according to an embodiment is schematically shown. DETAILED DESCRIPTION

[0059] In the following detailed description, in order to provide a thorough understanding of the relevant teachings, examples of a large number of details are listed. However, it is obvious to those skilled in the art that the present teachings can be practiced without these details. In other cases, known methods, steps and components have been described at a relatively high level without detailed description to avoid unnecessary obscurity of various aspects of the present teachings.

[0060] The three-dimensional coordinate system using the right-hand rule is defined in Section A1.1 of the Appendix. The two-dimensional coordinate system in the camera image frame is defined in Section A1.2 of the Appendix.

[0061] Mathematical utility functions used in this disclosure are listed in Section A2 of the Appendix. Quaternions, vectors, and matrix mathematics are discussed here. Quaternions are widely used in this disclosure. Functions using quaternions can also be expressed using matrices, Euler angles, or other suitable mathematical expressions.

[0062] Abbreviations of the form "ABC" are used in this disclosure: A stands for type; B stands for specific; C stands for context. For example, the abbreviation "qch" means: the quaternion "q" representing the orientation of the camera "c" in the head coordinate system "h". See Section A2 of the Appendix.

[0063] The spatial orientation sensor can measure its own orientation in three-dimensional space. In one embodiment, two spatial orientation sensors GE and GF are mounted on a frame so that their relative orientation remains unchanged. During the calibration process, the orientation of the spatial orientation GE relative to the spatial orientation GF can be obtained using the method described in the present disclosure (e.g., see section C5 of the method section). Since spatial orientation sensors can be attached to other objects to measure their orientation in three-dimensional space, the relative orientation between the two objects can be measured by attaching two spatial orientation sensors to the two objects respectively.

[0064] A camera can be used as an imaging sensor. In the present invention, the term "imaging sensor" and the term "camera" can be used interchangeably. In one embodiment, a spatial orientation sensor is attached to the camera. The camera and the spatial orientation sensor can be mounted on a rigid frame, such as Figure 5 As shown, their relative orientation does not change. The relative orientation of the camera and the spatial orientation sensor can be obtained using the system and method described in method section C9.

[0065] In one embodiment, there may be two cameras CE and CF, each with a spatial orientation sensor attached. The spatial orientation sensor attached to the camera CE is called GE. The spatial orientation sensor attached to the CF camera is called GF. The relative orientation of the two cameras CE and CF remains unchanged. Therefore, for example, the system and method described in part C10 of the method section can be used to obtain the orientation of the camera CE relative to the camera CF based on the orientation of the camera CE relative to the spatial orientation sensor GE, the orientation of the camera CF relative to the spatial orientation sensor GF, and the orientation of the spatial orientation sensor GE relative to the spatial orientation sensor GF. The system and method can be used in a wearable eye tracking system with multiple imaging sensors, multiple spatial orientation sensors, a computing device, and a storage device to detect and record where the user is looking. In an embodiment, the camera CE is set to face the user's eyes, called an "eye camera"; the camera CF is set to face away from the eyes, called a "field of view camera." As Figure 6As shown, the eye camera CE takes pictures of the eyes, and the field of view camera CF takes pictures of the scene in front of the user.

[0066] The gaze vector is a unit vector pointing from the center of the eye to the gaze point, which is the point the user is looking at. The gaze vector represents the orientation of the eye.

[0067] Assuming that the gaze vector in the eye camera CE coordinate system is known, the gaze vector can be converted to the field camera CF coordinate system using the method in section C12 of the method section. The gaze vector converted in the field camera CF coordinate system can be further converted to a point on the two-dimensional image plane of the field camera CF using the method in section C13 of the method section, so that the direction the user is looking at can be marked.

[0068] The systems and methods described in the present disclosure can also be used in other systems that use multiple cameras and multiple spatial orientation sensors, such as a panoramic camera system that acquires images from multiple cameras at a time and stitches them into a single image. In one embodiment, camera CE and camera CF are set to face different orientations, and there is a certain degree of overlap in their fields of view. The images captured by the two cameras can be stitched based on the relative orientation of the two cameras. In one embodiment, based on the relative orientation between the two cameras, the image of the point in the image plane of camera CE can be determined based on the image of the point in the three-dimensional space in the image plane of camera CF, and vice versa. For example, when a user specifies a point in an image captured by one camera, the corresponding point in the image captured by another camera can be obtained. Using the system and method described in the present disclosure, two or more cameras and two or more spatial orientation sensors can be calibrated.

[0069] During the calibration process, using the systems and methods described in this disclosure (eg, see Section C11 of the Methods section), the camera focal length can be obtained.

[0070] Each camera and each spatial orientation sensor has its own coordinate system, such as Figure 4 They are:

[0071] The camera CE coordinate system is Xce-Yce-Zce-Oce, referred to as CS-CE;

[0072] Camera CF coordinate system Xcf-Ycf-Zcf-Ocf, referred to as CS-CF;

[0073] The spatial orientation sensor GE coordinate system is Xge-Yge-Zge-Oge, referred to as CS-GE;

[0074] The spatial orientation sensor GF coordinate system is Xgf-Ygf-Zgf-Ogf, referred to as CS-GF;

[0075] About GE's world coordinate system Xwe-Ywe-Zwe-Owe, referred to as CS-WE;

[0076] About GF's world coordinate system Xwf-Ywf-Zwf-Owf, abbreviated as CS-WF.

[0077] The universal world coordinate system is Xw-Yw-Zw-Ow, referred to as CS-W. The universal camera coordinate system is Xc-Yc-Zc-Oc, referred to as CS-C. The universal spatial orientation sensor coordinate system is Xg-Yg-Zg-Og, referred to as CS-G.

[0078] Various hardware components can be used to implement the functions and processes in the present disclosure. One hardware component is a camera. A camera can measure the brightness of light, or brightness and color. In the present disclosure, the term "camera coordinate system" or "CS-C" is used interchangeably with the term "imaging sensor coordinate system" or "CS-I". A camera can capture color images or grayscale images, and can also capture infrared images or non-infrared (such as visible light) images. The parameters of a camera include its physical size, resolution, and the focal length of the lens. The two-dimensional camera image frame coordinate system is defined in Appendix Section A1.2.

[0079] A hardware component may also be a head device that can fix the camera on the user's head. The head device may be a spectacle frame, a headband or a helmet, depending on the application. Other forms of head devices are also possible.

[0080] A hardware component can be a computer, such as an embedded system or a desktop system.

[0081] exist Figure 6 In the embodiment of the eye tracking and gaze point recording system shown in FIG. 1 , an eye camera and a field of view camera are attached to a fixed frame worn by the user. The relative position between the two cameras does not change. The position of the two cameras relative to the user's head does not change. Figure 5 As shown, each camera is attached with a spatial orientation sensor. The orientation of the camera relative to the spatial orientation sensor remains unchanged.

[0082] The following describes how to obtain the relative orientation of the reference coordinate system with respect to the spatial orientation sensor according to two rotation vectors.

[0083] The spatial orientation sensor can be attached to a physical object to measure its orientation relative to the spatial orientation sensor. The orientation of the reference coordinate system CS-R in the spatial orientation sensor coordinate system CS-G can be described by a quaternion. Based on this quaternion, the coordinate system CS-R can be uniquely determined. For example, as described in Section C1 or Section C3 of the Methods section, this quaternion can be determined by two vectors in the spatial orientation sensor coordinate system. For example, as described in Section C2 or Section C4 of the Methods section, these two vectors can be obtained by two orientation outputs of the spatial orientation sensor.

[0084] The following describes how to obtain the relative positions of two spatial position sensors.

[0085] When two spatial orientation sensors are mounted on a frame and their relative orientations remain unchanged, the relative orientations of the two spatial orientation sensors can be obtained, for example, using the method described in section C5 of the method section. This can be achieved by finding the relative orientation of the common reference coordinate system relative to the spatial orientation sensor, and obtaining the relative orientations of the two spatial orientation sensors based on the relative orientation.

[0086] The following describes how to obtain the relative orientation of the reference coordinate system with respect to the camera based on an image of a pair of fixed points in the camera's field of view in three-dimensional space.

[0087] The orientation of the reference coordinate system CS-AB in the camera coordinate system CS-C can be described by a quaternion. Based on this quaternion, the coordinate system CS-AB can be uniquely determined. For example, as described in Section C1 of the Methods section, this quaternion can be determined from two vectors in the camera coordinate system. As described in Section C6 of the Methods section, these two vectors can be obtained using an image of a pair of fixed points in the three-dimensional space in the camera's field of view.

[0088] The following describes how to obtain the rotation vector based on the orientation change of the reference coordinate system in the camera coordinate system.

[0089] As mentioned above, the relative orientation of the reference coordinate system with respect to the camera can be obtained by the image of a pair of fixed points in the three-dimensional space in the camera's field of view. By repeating this process at two different camera orientations, two orientations of the reference coordinate system with respect to the camera coordinate system can be obtained. Based on these two orientations, the rotation vector in the camera coordinate system can be determined, as described in Section C7 or Section C2 of the Methods section.

[0090] The following describes how to obtain the relative orientation of another reference coordinate system relative to the camera based on two rotation vectors.

[0091] As described above, the rotation vector in the camera coordinate system can be determined by using a pair of fixed points in the three-dimensional space in the camera field of view at two different camera orientations. For example, as described in section C8 of the method section, two rotation vectors can be obtained by using two different pairs of fixed points in the three-dimensional space in the camera field of view. Based on these two rotation vectors, the orientation of another reference coordinate system relative to the camera coordinate system can be obtained, for example, using the method described in section C1 of the method section.

[0092] The following describes how to obtain the position of the camera relative to the spatial position sensor.

[0093] For example, when the camera and the spatial orientation sensor are mounted on a rigid frame, assuming that the orientation of the camera relative to the spatial orientation sensor is unchanged, the orientation of the reference coordinate system relative to the spatial orientation sensor can be determined based on the two rotation vectors, and as described above, the orientation of another reference coordinate system relative to the camera can be determined based on the two rotation vectors. Because the orientation of the camera relative to the spatial orientation sensor is unchanged, if these reference frames are obtained simultaneously based on the same orientation change, then these reference frames are the same, and the orientation of the camera relative to the spatial orientation sensor can be obtained using the method described in Section C9 of the Methods section.

[0094] The following describes how to obtain the relative positions of the two cameras.

[0095] If the relative orientation of the two cameras does not change (for example, when they are mounted on a rigid frame), and assuming that each camera is rigidly attached to the spatial orientation sensor, then the relative orientation of the two spatial orientation sensors also does not change. As described above, the orientation of each camera relative to the spatial orientation sensor to which it is attached can be determined, and the relative orientation of the two spatial orientation sensors can be determined. The relative orientation between the two cameras can be obtained using the method described in Section C9 of the Methods section.

[0096] The following describes how to obtain the focal length of the camera.

[0097] The focal length of the camera can be used to calculate the reference coordinate system and rotation vector, see Section C6 or Section C7 of the Methods section. The focal length can be obtained using the method described in Section C11 of the Methods section.

[0098] Given a focal length value, the rotation vector and rotation amount can be obtained at two camera orientations using the method described in Method Section C7. In this process, two orientation outputs from a spatial orientation sensor rigidly attached to the camera can also be obtained. Using the method described in Section C2 or Section C4 of the Method Section, a rotation vector and rotation amount can be obtained based on the two orientation outputs. This rotation vector and rotation amount can be compared with the rotation vector and rotation amount obtained in the same process using a set of known focal lengths, and the difference between them can be used to estimate the focal length of the camera.

[0099] The following describes the conversion of a vector from one camera's coordinate system to another camera's coordinate system.

[0100] If the relative position of one camera with respect to the other camera is known, a vector in the coordinate system of one camera can be converted to a vector in the coordinate system of the other camera, for example, as described in Section C12 or Section C13 of the Methods section.

[0101] The following describes the conversion of the gaze vector between the eye camera and the field of view camera.

[0102] As described above, in a system where the eye camera CE faces the user's eyes and the field of view camera CF faces the outside world, the gaze vector can be transformed from the coordinate system of the eye camera CE to the coordinate system of the field of view camera CF, and vice versa. The gaze vector in the coordinate system of the field of view camera CF can be mapped to a point on the image plane of the field of view camera CF.

[0103] The following describes the transformation of points from the image plane of one camera to the image plane of another camera.

[0104] Vectors in the camera coordinate system can be mapped to points on the camera's image plane. See section A1.3 of the Appendix. Points on a camera's image plane can be mapped to its coordinate system. See section A1.4 of the Appendix. Given the relative orientation of the cameras, points on one camera's image plane can be mapped to the image plane of another camera. This can be used to stitch images acquired by multiple cameras.

[0105] Fig. 7A A flow chart of a method according to another embodiment is shown. In step 710, a first orientation 711, a second orientation 712, a third orientation 713, and a fourth orientation 714 of a first spatial orientation sensor, and a first orientation 715, a second orientation 716, a third orientation 717, and a fourth orientation 718 of a second spatial orientation sensor are acquired. In step 720, a relative orientation 721 of the first spatial orientation sensor relative to the second spatial orientation sensor is determined based on the first orientation 711, the second orientation 712, the third orientation 713, and the fourth orientation 714 of the first spatial orientation sensor, and the first orientation 715, the second orientation 716, the third orientation 717, and the fourth orientation 718 of the second spatial orientation sensor. The relative orientation 721 of the first spatial orientation sensor relative to the second spatial orientation sensor is fixed.

[0106] Figure 7BAn example of the details of step 720 is schematically shown. In step 7210, a relative orientation 7211 of the first spatial orientation sensor relative to the reference system is determined. In step 7220, a relative orientation 7221 of the second spatial orientation sensor relative to the reference system is determined. In one embodiment, the relative orientation 7211 of the first spatial orientation sensor relative to the reference system in step 7210 is determined based on the first orientation 711, the second orientation 712, the third orientation 713, and the fourth orientation 714. In an embodiment, the relative orientation 7221 of the second spatial orientation sensor relative to the reference system in step 7220 is determined based on the first orientation 715, the second orientation 716, the third orientation 717, and the fourth orientation 718. In step 7230, the relative orientation 721 is determined based on the relative orientation 7211 of the first spatial orientation sensor relative to the reference system and the relative orientation 7221 of the second spatial orientation sensor relative to the reference system.

[0107] Fig. 8A A flow chart of a method according to another embodiment is schematically shown. In step 810, an image 811 of a first point in three-dimensional space and an image 812 of a second point in three-dimensional space are acquired using an imaging sensor. In step 820, a relative orientation 821 of the imaging sensor relative to a reference system is determined based on the image 811 and the image 812. The first point and the second point are fixed in three-dimensional space.

[0108] Figure 8B An example of the details of step 820 is schematically shown. In step 8210, a vector 8211 is determined between a reference point fixed relative to the imaging sensor and the image 811 of the first point. In step 8220, a vector 8221 is determined between the reference point and the image 822 of the second point. In step 8230, a relative orientation 821 is determined based on the vector 8211 and the vector 8221.

[0109] Fig. 9A A flow chart of a method according to another embodiment is schematically shown. In step 910, a first image 991 of a first point in a three-dimensional space and a second image 992 of a second point in the three-dimensional space are acquired using an imaging sensor in its first orientation. In step 920, a third image 993 of the first point and a fourth image 994 of the second point are acquired using the imaging sensor in its second orientation. In step 930, a relative rotation 931 of the imaging sensor between the first orientation and the second orientation is obtained based on the first image 991, the second image 992, the third image 993 and the fourth image 994. The first point and the second point are fixed in the three-dimensional space.

[0110] Fig. 9BAn example of the details of step 930 is schematically shown. In step 9310, a relative orientation 9311 of the imaging sensor in the first orientation relative to the reference system is determined based on the first image 991 and the second image 992. In step 9320, a relative orientation 9321 of the imaging sensor in the second orientation relative to the reference system is determined based on the third image 993 and the fourth image 994. In step 9330, a relative rotation 931 is determined based on the relative orientation 9311 and the relative orientation 9321.

[0111] Fig. 10A A flow chart of a method according to another embodiment is schematically shown. In step 1010, a first relative rotation 1011 of the imaging sensor between a first orientation of the imaging sensor and a second orientation of the imaging sensor is determined. In step 1020, a second relative rotation 1021 of the imaging sensor between a third orientation of the imaging sensor and a fourth orientation of the imaging sensor is determined. In step 1030, a relative orientation 1031 of the imaging sensor relative to a reference system is determined based on the first relative rotation 1011 and the second relative rotation 1021.

[0112] Fig. 10B An example of details in step 1010 is schematically shown. In step 10110, a first image 10111 of a first point in a three-dimensional space and a second image 10112 of a second point in the three-dimensional space are acquired using an imaging sensor in a first orientation. In step 10120, a third image 10121 of the first point and a fourth image 10122 of the second point are acquired using an imaging sensor in a second orientation. In step 10130, a first relative rotation 1011 is determined based on the first image 10111, the second image 10112, the third image 10121, and the fourth image 10122.

[0113] Fig.11AA flow chart of a method according to another embodiment is schematically shown. In optional step 1100, a first relative rotation 1101 of the imaging sensor between a first orientation of the imaging sensor and a second orientation of the imaging sensor, and a second relative rotation 1102 of the imaging sensor between a third orientation of the imaging sensor and a fourth orientation of the imaging sensor are determined. In step 1110, a relative orientation 1111 of the imaging sensor relative to a reference system is determined based on the first relative rotation 1101 and the second relative rotation 1102. In step 1120, a first output 1121, a second output 1122, a third output 1123, and a fourth output 1124 corresponding to the spatial orientation sensor are determined when the imaging sensor is in the first orientation, the second orientation, the third orientation, and the fourth orientation, respectively. In step 1130, a relative orientation 1131 of the spatial orientation sensor relative to the reference system is determined based on the first output 1121, the second output 1122, the third output 1123, and the fourth output 1124. At step 1140 , a relative position 1141 of the spatial position sensor relative to the imaging sensor is determined based on the relative position 1131 of the spatial position sensor relative to the reference system and the relative position 1111 of the imaging sensor relative to the reference system.

[0114] Fig. 11B An example of details in optional step 1100 is schematically shown. In step 11110, a first image 11111 of a first point in three-dimensional space and a second image 11112 of a second point in three-dimensional space are acquired using an imaging sensor in a first orientation. In step 11120, a third image 11121 of the first point and a fourth image 11122 of the second point are acquired using an imaging sensor in a second orientation. In step 11130, a first relative rotation 1101 is determined based on the first image 11111, the second image 11112, the third image 11121, and the fourth image 11122.

[0115] Fig.12A flow chart of a method according to another embodiment is schematically shown. In step 1210, a first relative orientation 1211 of the first spatial orientation sensor relative to the first imaging sensor is determined based on a relative orientation 1201 of the first spatial orientation sensor relative to the first reference system and a relative orientation 1291 of the first imaging sensor relative to the first reference system. In step 1220, a second relative orientation 1221 of the second spatial orientation sensor relative to the second imaging sensor is determined based on a relative orientation 1202 of the second spatial orientation sensor relative to the second reference system and a relative orientation 1292 of the second imaging sensor relative to the second reference system. In step 1230, a third relative orientation 1231 of the first spatial orientation sensor relative to the second spatial orientation sensor is determined. In step 1240, a fourth relative orientation 1241 of the first imaging sensor relative to the second imaging sensor is determined based on the first relative orientation 1211, the second relative orientation 1221 and the third relative orientation 1231.

[0116] Fig.13 A flowchart of a method according to another embodiment is schematically shown. In step 1310, a first image 1311 of a first point in a three-dimensional space and a second image 1312 of a second point in the three-dimensional space are acquired using an imaging sensor at its first orientation. In step 1320, a third image 1321 of the first point and a fourth image 1322 of the second point are acquired using an imaging sensor at its second orientation. In step 1330, a first output 1331 and a second output 1332 of the spatial orientation sensor are acquired when the imaging sensor is in the first orientation and the second orientation, respectively. In step 1340, a focal length 1341 of the imaging sensor is determined based on the first image 1311, the second image 1312, the third image 1321, the fourth image 1322, the first output 1331 and the second output 1332.

[0117] Fig.14 A flowchart of a method according to another embodiment is schematically shown. In step 1410, a first output 1411, a second output 1412, a third output 1413, and a fourth output 1414 of the spatial orientation sensor are obtained when the spatial orientation sensor is in a first orientation, a second orientation, a third orientation, and a fourth orientation, respectively. In step 1420, based on the first output 1411, the second output 1412, the third output 1413, and the fourth output 1414, a relative orientation 1421 of the spatial orientation sensor relative to the reference system is determined.

[0118] Fig.15The flowchart of the method according to another embodiment is schematically shown. In step 1510, based on the image acquired using the first imaging sensor, a first relative position 1511 of the eye of the person relative to the first imaging sensor is acquired. In step 1520, a second relative position 1521 of the second imaging sensor relative to the first imaging sensor is acquired. In step 1530, based on the first relative position 1511 and the second relative position 1512, a third relative position 1531 of the eye relative to the second imaging sensor is determined.

[0119] method

[0120] C1

[0121] In the known coordinate system CS-A, obtain the first three-dimensional unit vector v_a_1 and the second three-dimensional unit vector v_a_2; and determine the orientation of the reference coordinate system CS-R relative to the coordinate system CS-A.

[0122] This can be expressed as:

[0123] q_r_a=q_frm_2v(v_a_1,v_a_2,method_index)

[0124] Where method_index indicates the method of obtaining the unique orientation quaternion q_r_a of CS-R in CS-A based on v_a_1 and v_a_2. The coordinate system CS-R is uniquely determined by q_r_a.

[0125] There are two ways to get q_r_a:

[0126] method_index = 1;

[0127] q_r_a=q_frm_m(m)

[0128] in:

[0129] m = m_frm_v(v_x,v_y,v_z), see Appendix Section A2.3.10;

[0130] in:

[0131] v_x=v_a_1,v_z=v_cru(v_x,v_a_2),v_y=v_cru(v_z,v_x), see Appendix Section A2.2.8;

[0132] in:

[0133] m is a 3x3 matrix, v_a_1 is the first unit vector in CS-A, and v_a_2 is the second unit vector in CS-A.

[0134] method_index=2:

[0135] q_r_a=q_frm_m(m)

[0136] in:

[0137] m = m_frm_v(v_x,v_y,v_z), see Appendix Section A2.3.10;

[0138] in:

[0139] vx=v_cru(v_y,v_z);

[0140] in:

[0141] vy=v_cru(v_a_2,v_a_1)

[0142] vz=v_neg(v_uni(v_add(v_a_1,v_a_2))), see Appendix chapters A2.2.10, A2.2.3, and A2.2.11.

[0143] Other methods can also be used to obtain the unique orientation quaternion q_r_a of CS-R in CS-A based on the two three-dimensional vectors v_a_1 and v_a_2 in CS-A.

[0144] C2

[0145] Gets the vector v_x that is aligned with the rotation axis around which the coordinate system CS-X rotates from the starting quaternion q_x_w_1 relative to CS-W to the ending quaternion q_x_w_2 relative to CS-W. Also gets the amount of rotation in degrees.

[0146] This can be described as:

[0147] v_x=v_frm_2q(q_x_w_1,q_x_w_2)

[0148] deg=deg_frm_2q(q_x_w_1,q_x_w_2)

[0149] in:

[0150] q_x_w_1 is the orientation quaternion representing the starting orientation of CS-X in CS-W.

[0151] q_x_w_2 is the orientation quaternion representing the ending orientation of CS-X in CS-W.

[0152] CS-Y is defined as the coordinate system in CS-W that is aligned with CS-X at the starting orientation q_x_w_1. Assuming q_x_y_1 is the quaternion of q_x_w_1 transformed from CS-W to CS-Y, then: q_x_y_1 = q_ldf(q_x_w_1,q_x_w_1). Assuming q_x_y_2 is the quaternion of q_x_w_2 transformed from CS-W to CS-Y, then: q_x_y_2 = q_ldf(q_x_w_1,q_x_w_2). Here, q_x_y_1 is the unit quaternion, and q_x_y_2 represents the orientation change from q_x_y_1 to q_x_y_2 in CS-Y. See Appendix Section A2.1.9.

[0153] Assuming v_x is the rotation vector that can be obtained from q_x_y_2, v_x = v_frm_q(q_x_y_2), then v_x can be regarded as the axis of rotation of CS-X from q_x_y_1 to q_x_y_2 in CS-Y. See Appendix Section A2.2.7.

[0154] Assume that deg is the rotation amount of the rotation vector v_x in degrees, deg = deg_frm_q(q_x_y_2), because CS-Y is aligned with the starting orientation of CS-X in CS-W, v_x can also be regarded as the rotation vector v_x in CS-X about which CS-X rotates from its starting orientation to its ending orientation. See Appendix Section A2.2.9.

[0155] C3

[0156] Gets the rotation vector aligned with the rotation axis of the spatial orientation sensor from the starting orientation to the ending orientation, and the amount of rotation in degrees.

[0157] The spatial orientation is the three-dimensional orientation of the coordinate system CS-G measured in the world coordinate system CS-W. Assumptions:

[0158] q_g_w_1 is the starting position of CS-G in CS-W,

[0159] q_g_w_2 is the ending position of CS-G in CS-W.

[0160] The rotation vector v_g can be obtained using the method described in Section C2 of the Methods section, where:

[0161] v_g can also be regarded as the rotation vector of the spatial orientation sensor in CS-G from its starting orientation to its ending orientation. The rotation amount expressed in degrees can also be obtained.

[0162] Therefore, v_g and deg_g can be obtained as described in Section C2 of the Methods section:

[0163] v_g=v_frm_2q(q_g_w_1,q_g_w_2)

[0164] deg_g=deg_frm_2q(q_g_w_1,q_g_w_2).

[0165] C4

[0166] Through the four orientation outputs of the spatial orientation sensor, the orientation quaternion q_r_g representing the reference coordinate system CS-R relative to the spatial orientation sensor coordinate system CS-G is obtained.

[0167] This can be described as:

[0168] q_r_g=q_frm_4q(q_g_w_1,q_g_w_2q_g_w_3,q_g_w_4,method_index)

[0169] in:

[0170] q_g_w_1, q_g_w_2, q_g_w_3, q_g_w_4 are the outputs of the spatial orientation sensors at four locations relative to CS-W. The four locations do not have to be completely different as long as the following three conditions are met:

[0171] (1) At least three of these four quaternions are different;

[0172] (2) q_g_w_1 is different from q_g_w_2;

[0173] (3)q_g_w_3 and q_g_w_4 are different.

[0174] q_r_g=q_frm_2v(v_g_1, v_g_2, method_index), where method_index represents one of many different methods that can be used in part C1 in the methods section.

[0175] As described in Section C3 of the Methods section, v_g_1 = v_frm_2q(q_g_w_1, q_g_w_2); As described in Section C3 of the Methods section, v_g_2 = v_frm_2q(q_g_w_3, q_g_w_4).

[0176] C5

[0177] The quaternion q_ge_gf is obtained from the four orientations, and q_ge_gf represents the orientation of the spatial positioning sensor GE located in the CS-GE coordinate system relative to the spatial positioning sensor GF located in the CS-GF coordinate system.

[0178] This can be described as:

[0179] q_ge_gf=q_frm_8q(q_ge_we_1q_ge_we_2, q_ge_we_3q_ge_we_4, q_gf_wf_1, q_gf_wf_2, q_gf_wf_3, q_gf_wf_4, method_index).

[0180] Assume that there are two spatial positioning sensors GE and GF, where

[0181] CS-GE is the coordinate system of the spatial positioning sensor GE, and CS-GF is the coordinate system of the spatial orientation sensor GF. The two spatial orientation sensors are installed on a frame. The relative orientation does not change during the calibration process. In the orientation of a given frame, the outputs of the two spatial orientation sensors are:

[0182] Quaternion q_ge_we_n, indicating the position of CS-GE in CS-WE

[0183] Quaternion q_gf_wf_n represents the orientation of CS-GF in CS-WF, where n can be 1, 2, 3, or 4.

[0184] CS-WE is the world coordinate system referenced by the spatial orientation sensor output. CS-WF is the world coordinate system referenced by the spatial orientation sensor GF output.

[0185] The output quaternions of the two spatial orientation sensors are grouped in pairs in the form of (q_ge_we_n,q_gf_wf_n), where n can be 1, 2, 3, or 4. For each q_ge_we_n output of spatial orientation sensor E, there is also a q_gf_wf_n output of spatial orientation sensor F.

[0186] Call q_gf_wf_n as the paired quaternion of q_ge_we_n, where n can be 1, 2, 3, or 4:

[0187] q_ge_we_1, q_ge_we_2, q_ge_we_3, q_ge_we_4 are relative to CS-WE and do not have to be completely different as long as the following three conditions are met:

[0188] (1) At least three of these four quaternions are different;

[0189] (2) q_ge_we_1 is different from q_ge_we_2;

[0190] (3)q_ge_we_3 and q_ge_we_4 are different.

[0191] q_gf_wf_1, q_gf_wf_2, q_gf_wf_3, q_gf_wf_4 are relative to CS-WF. As long as the following three conditions are met, q_gf_wf_3 and q_gf_wf_4 are not necessarily different:

[0192] (1) At least three of these four quaternions are different;

[0193] (2) q_gf_wf_1 is different from q_gf_wf_2; and

[0194] (3)q_gf_wf_3 and q_gf_wf_4 are different.

[0195] Use the method described in Methods section C4:

[0196] q_re_ge=q_frm_4q(q_ge_we_1, q_ge_we_2, q_ge_we_3, q_ge_we_4, method_index)

[0197] q_rf_gf=q_frm_4q(q_gf_wf_1, q_ge_wf_2, q_gf_wf_3, q_gf_wf_4, method_index)

[0198] in:

[0199] method_index is one of many different methods that can be used in the C1 section of the methods chapter;

[0200] q_re_ge is the quaternion representing the orientation of CS-RE in CS-GE;

[0201] q_rf_gf is the quaternion representing the orientation of CS-RF in CS-GF;

[0202] CS-RE is a reference coordinate system that can be uniquely determined using q_ge_we_1, q_ge_we_2, q_ge_we_3, q_ge_we_4 and method_index;

[0203] CS-RF is a reference coordinate system that can be uniquely determined using q_gf_wf_1, q_gf_wf_2, q_gf_wf_3, q_gf_wf_4, and method_index.

[0204] Since CS-RE and CS-RF can be regarded as the same coordinate system, the relative orientation of CS-GE and CS-GF can be expressed as:

[0205] q_ge_gf=q_prd2(q_rf_gf,q_cnj(q_re_ge)).

[0206] C6

[0207] Get the orientation quaternion q_ab_c from the images p_a_i and p_b_i, such as Figure 2 As shown, the pair of points PA and PB are fixed in the world coordinate system CS-W, and the camera coordinate system CS-C is also in CS-W. The two fixed points are in the field of view of the camera. q_ab_c represents the orientation of the reference coordinate system CS-AB in the camera coordinate system CS-C.

[0208] This can be described as:

[0209] q_ab_c=q_frm_2p(p_a_i, p_b_i, FOCAL_LEN, method_index).

[0210] As described in Section A1.2, assume that there is a pair of fixed physical points PA and PB in the world coordinate system CS-W, which are in the field of view of the camera, p_a_i and p_b_i are the images of PA and PB in the camera image plane, FOCAL_LEN is the focal length of the camera in pixels, and v_c_a and v_c_b are the unit vectors pointing from the origin of CS-C to p_a_i and p_b_i, respectively. You can use the method in Section A1.3 or Section A1.4 of the Appendix to get v_a_c and v_b_c based on p_a_i and p_b_i:

[0211] v_a_c=v_frm_p(p_a_i,FOCAL_LEN);

[0212] v_b_c=v_frm_p(p_b_i,FOCAL_LEN).

[0213] Knowing these two three-dimensional vectors v_a_c and v_b_c in CS-C, the reference coordinate system CS-AB can be determined. The quaternion q_ab_c representing the orientation of CS-AB in CS-C can be obtained using the method described in part C1 of the method section:

[0214] q_ab_c=q_frm_2v(v_a_c, v_b_c, method_index), where:

[0215] method_index represents one of many different methods that can be used in the C1 section.

[0216] C7

[0217] Using the images of a pair of fixed physical points PA and PB obtained by the camera at two different orientations, the rotation vector in the camera coordinate system CS-C is obtained, and the rotation amount in degrees is obtained at the same time.

[0218] This can be expressed as:

[0219] v_c=v_frm_4p(p_a_i_1,p_b_i_1,p_a_i_2,p_b_i_2,FOCAL_LEN,method_index);

[0220] deg_c=deg_frm_4p(p_a_i_1, p_b_i_1, p_a_i_2, p_b_i_2, FOCAL_LEN, method_index).

[0221] Using the same pair of physical points PA and PB in the world coordinate system CS-W (where CS-C is also located), the steps of section C6 can be performed twice using the camera at its two different orientations.

[0222] Assumptions:

[0223] p_a_i_1 and p_b_i_1 are the images of physical points PA and PB captured by the camera at the first orientation;

[0224] p_a_i_2 and p_a_i_2 are images of physical points PA and PB captured by the camera at the second orientation; and

[0225] FOCAL_LEN is the focal length of the camera, in pixels.

[0226] Use the method described in Methods section C6:

[0227] q_ab_c_1=q_frm_2p(p_a_i_1, p_b_i_1, FOCAL_LENmethod_index),

[0228] q_ab_c_2=q_frm_2p(p_a_i_2, p_b_i_2, FOCAL_LENmethod_index);

[0229] in:

[0230] method_index is one of many different methods that can be used in the C1 section of the methods chapter;

[0231] q_ab_c_1 is the position of CS-AB when the camera in CS-C is in the first position;

[0232] q_ab_c_2 is the position of CS-AB when the camera in CS-C is in the second position.

[0233] CS-AB is the reference coordinate system determined by the images of the two physical points PA and PB as described in Section C6 of the Methods chapter.

[0234] Given q_ab_c_1 and q_ab_c_2, the rotation vector v_c and the rotation angle can be obtained using the method described in C2 in the method section:

[0235] v_c=v_frm_2q(q_ab_c_1,q_ab_c_2);

[0236] deg_c=deg_frm_2q(q_ab_c_1,q_ab_c_2);

[0237] Among them, v_c can be regarded as the rotation vector of the camera from the starting orientation to the ending orientation in the coordinate system aligned with the starting orientation of CS-C.

[0238] C8

[0239] Use the images of four fixed physical points PA, PB, PC, and PD taken by the camera at its four orientations to obtain the orientation of the reference coordinate system CS-RC relative to the camera coordinate system CS-C.

[0240] This can be expressed as:

[0241] q_rc_c=q_frm_8p(p_a_i_1,p_b_i_1,p_a_i_2,p_b_i_2,p_c_i_1,p_d_i_1,p_c_i_2,p_b_i_d,FOCAL_LEN,method_index).

[0242] The process described in Section C7 can be performed twice. One physical point is selected each time. Thus:

[0243] p_a_i_1 and p_b_i_1 are the images of physical points PA and PB captured by the camera at the orientation defined as AB1;

[0244] p_a_i_2 and p_b_i_2 are the images of physical points PA and PB captured by the camera at the orientation defined as AB2;

[0245] p_c_i_1 and p_d_i_1 are the images of physical points PC and PD captured by the camera at the position defined as CD1;

[0246] p_c_i_2 and p_d_i_2 are the images of physical points PC and PD captured by the camera at the position defined as CD2;

[0247] FOCAL_LEN is the focal length of the camera in pixels.

[0248] The four camera positions AB1, AB2, CD1, and CD2 do not have to be completely different as long as the following three conditions are met:

[0249] (1) At least three of the four directions are different;

[0250] (2) Camera position numbers AB1 and AB2 are different;

[0251] (3) The camera position numbers CD1 and CD2 are different.

[0252] The first pair of physical points is PA and PB. The second pair of physical points is PC and PD. The second pair of physical points PC and PD can be the same as or different from PA and PB. Therefore, using the method described in Method Section C7:

[0253] v_ab_c=v_frm_4p(p_a_i_1,p_b_i_1,p_a_i_2,p_b_i_2,FOCAL_LEN,method_index)

[0254] v_cd_c=v_frm_4p(p_c_i_1,p_d_i_1,p_c_i_2,p_d_i_2,FOCAL_LEN,method_index)

[0255] in:

[0256] method_index is one of many different methods that can be used in C1 in the methods chapter.

[0257] The orientation quaternion q_rc_c represents the orientation of the reference coordinate system CS-RC in CS-C:

[0258] q_rc_c=q_frm_2v(v_ab_c,v_cd_c,method_index);

[0259] where method_index in the method section represents one of many different methods that can be used in Part C1.

[0260] C9

[0261] Using images of four fixed physical points captured by the camera at its four orientations and the spatial orientation sensor, an orientation quaternion q_c_g representing the orientation of the camera coordinate system CS-C relative to the spatial orientation sensor located in the coordinate system CS-G is obtained.

[0262] This can be described as:

[0263] q_cg=q_frm_qc_qg(q_rc_c,q_rg_g,FOCAL_LEN,method_index)

[0264] It is assumed that the camera and the spatial orientation sensor are mounted on a rigid frame. During the calibration process, their relative orientation between CS-C and CS-G does not change.

[0265] First, using the process described in the method of part C8, the quaternion q_rc_c representing the orientation of CS-RC in the CS-C reference coordinate system can be obtained. Specifically, this process requires the camera to capture a pair of physical points PA and PB in the three-dimensional world coordinate system CS-W where CS-C is located. The camera needs to capture images of this pair of fixed physical points PA and PB when it is located at the four orientations defined as AB1, AB2, CD1 and CD2.

[0266] q_rc_c=q_frm_8p(p_a_i_1,p_b_i_1,p_a_i_2,p_b_i_2,p_c_i_1,p_d_i_1,p_c_i_2,p_b_i_d,FOCAL_LEN,method_index)

[0267] in:

[0268] p_a_i_1 and p_b_i_1 are the images of physical points PA and PB captured by the camera at the AB1 position. p_a_i_2 and p_b_i_2 are the images of physical points PA and PB captured by the camera at the AB2 position. p_c_i_1 and p_d_i_1 are the images of physical points PC and PD captured by the camera at the CD1 position. p_c_i_2 and p_d_i_2 are the images of physical points PC and PD captured by the camera at the CD2 position. FOCAL_LEN is the focal length of the camera, in pixels.

[0269] During this process, the output of the spatial orientation sensor is also recorded:

[0270] q_g_w_ab1 is the spatial orientation sensor output at camera orientation AB1;

[0271] q_g_w_ab2 is the spatial orientation sensor output at camera orientation AB2;

[0272] q_g_w_cd1 is the spatial orientation sensor output at camera orientation CD1;

[0273] q_g_w_cd2 is the spatial orientation sensor output at camera orientation CD2.

[0274] The orientation quaternion q_rg_g represents the orientation of the reference coordinate system CS-RG in CS-G and can be obtained using the method described in Methods section C4:

[0275] q_rg_g=q_frm_4q(q_g_w_ab1, q_g_w_ab2, q_g_w_cd1, q_g_w_cd2, method_index).

[0276] Since CS-RC and CS-RG can be regarded as the same coordinate system, the orientation quaternion of CS-C in CS-G can be expressed as:

[0277] q_c_g=q_prd2(q_rg_g,q_cnj(q_rc_c)).

[0278] C10

[0279] Get the orientation quaternion q_ce_cf, where q_ce_cf represents the orientation of the coordinate system CS-CE of the camera CE relative to the coordinate system CS-CF of the camera CF.

[0280] This can be expressed as:

[0281] q_ce_cf=q_frm_3q(q_ce_ge,q_cf_gf,q_ge_gf,FOCAL_LEN,method_index)

[0282] Assumptions:

[0283] (1) The camera CE and the spatial orientation sensor GE are mounted on a rigid frame. Their relative orientation does not change during the calibration process.

[0284] (2) The camera CF and the spatial orientation sensor GF are mounted on a rigid frame. Their relative orientation does not change during the calibration process.

[0285] (3) The relative positions of the spatial orientation sensor GE and the spatial orientation sensor GF do not change during the calibration process:

[0286] Using the method described in Methods section C9, we can get the following orientation quaternion:

[0287] q_ce_ge is the orientation quaternion, which indicates the orientation of the coordinate system CS-CE of the camera CE in the coordinate system CS-GE of the spatial orientation sensor GE;

[0288] q_cf_gf is the orientation quaternion representing the camera CF coordinate system CS-CF in the spatial orientation sensor GF coordinate system.

[0289] Using the method described in Methods section C5, we can get the following orientation quaternion:

[0290] q_ge_gf is the orientation quaternion representing the coordinate system CS-GE of the spatial orientation sensor GE in the coordinate system CS-GF of the spatial orientation sensor GF.

[0291] Therefore, the quaternion q_ce_cf representing the CS-CE orientation of the camera CE in the CS-CF coordinate system of the camera CF is:

[0292] q_ce_cf=q_prd2(q_cnj(q_cf_gf),q_prd2(q_ge_gf q_ce_ge)).

[0293] C11

[0294] The focal length FOCAL_LEN of the camera is obtained from the images of a pair of fixed physical points PA and PB in the three-dimensional world coordinate system CS-W, where the camera coordinate system CS-C is also located in CS-W, and the output corresponding to the spatial orientation sensor when the camera is located in two orientations is obtained.

[0295] Assume that the camera and the spatial orientation sensor are mounted on a rigid frame. During the calibration process, the relative orientation between the camera coordinate system CS-C and the spatial orientation sensor coordinate system CS-G remains unchanged.

[0296] The camera needs to capture images of a pair of fixed physical points PA and PB at two orientations defined as AB1 and AB2.

[0297] Using the method described in section C7 of the Methods section, you can get the rotation angle value deg_c:

[0298] deg_c=deg_frm_4p(p_a_i_1,p_b_i_1,p_a_i_2,p_b_i_2,FOCAL_LEN,method_index);

[0299] p_a_i_1 and p_b_i_1 are the images of physical points PA and PB captured by the camera at AB1;

[0300] p_a_i_2 and p_b_i_2 are the images of physical points PA and PB captured by the camera at AB2;

[0301] FOCAL_LEN is the focal length of the camera in pixels.

[0302] During this process, the output of the spatial orientation sensor is also recorded:

[0303] q_g_w_ab1 is the spatial orientation sensor output at camera orientation AB1;

[0304] q_g_w_ab2 is the spatial orientation sensor output at camera orientation AB2.

[0305] Using the method described in the C2 method, the rotation angle value deg_g can be obtained:

[0306] deg_g=deg_frm_2q(q_g_w_ab1,q_g_w_ab2).

[0307] Assume focal_len_n, where n = 1, 2, 3, 4, ... is a known focal length value:

[0308] deg_c_n=deg_frm_4p(p_a_i_1,p_b_i_1,p_a_i_2,p_b_i_2,focal_len_n,method_index)

[0309] Where n = 1, 2, 3, 4...

[0310] The focal_len_n with the closest deg_c_n value to deg_g can be considered as the best estimate of the camera focal length FOCAL_LEN.

[0311] C12

[0312] Transform the gaze vector v_ce in the camera CE coordinate system CS-CE to the vector v_cf in the camera CF coordinate system CS-CF.

[0313] This can be expressed as:

[0314] v_cf=qvq_trans(q_ce_cf,v_ce)

[0315] in:

[0316] v_cf is the result of converting the vector v_ce from CS-CE to CS-CF.

[0317] C13

[0318] Convert the vector v_cf in the camera CF coordinate system to a two-dimensional vector p_cf on the two-dimensional image plane of the camera CF.

[0319] After obtaining v_cf using the method described in Methods section C12, it can be mapped to a point p_cf at the 2D image position of camera CF:

[0320] p_cf=p_frm_v(v_cf,p_cent,FOCAL_LEN)

[0321] in:

[0322] p_cent is the center point of the CF camera image plane;

[0323] FOCAL_LEN is the focal length of the camera CF, in pixels.

[0324] appendix

[0325] The Methods section uses the mathematical tools listed in the Appendix section.

[0326] A1. Coordinate system

[0327] A1.1 The three-dimensional coordinate system has three axes: X, Y, and Z. Figure 3 The order of the axes and the positive direction of rotation are determined using the right-hand rule.

[0328] Any two axes can form a plane. Therefore, in a three-dimensional coordinate system, there are three planes defined as P-XY, P-YX and P-ZX.

[0329] A1.2 As Figure 1 As shown, there are two axes, X and Y, in the two-dimensional coordinate system of the camera image frame.

[0330] A1.3 Convert points in the two-dimensional camera image frame coordinate system to the three-dimensional camera coordinate system.

[0331] A three-dimensional camera coordinate system CS-C has the x-axis pointing to the right, the y-axis pointing to the top, and the z-axis pointing in the opposite direction of the camera.

[0332] The two-dimensional image plane can be considered as:

[0333] - Parallel to the XY plane of CS-C;

[0334] - The origin is in the upper left corner;

[0335] -The center of its image is at (0, 0, -1) of CS-C

[0336] -The center of its image is located at (0, 0, 1) of CS-C;

[0337] - Its X-axis is parallel to the X-axis of CS-C and points in the same direction;

[0338] - Its Y axis is parallel to the Y axis of CS-C and points in the opposite direction;

[0339] -Different units from CS-C. Specifically, FOCAL_LEN is the focal length of the camera in pixels.

[0340] Calculate the unit vector vu from the origin of CS-C to point p on the two-dimensional plane of the camera image:

[0341] vu=v_frm_p(p)

[0342] in,

[0343] p=(x,y)

[0344] vu=(vx,vy,vz)

[0345] in,

[0346] vu=v_uni(v)

[0347] v = (h, v, -1.0)

[0348] in,

[0349] h=DEFOX(x)=(x-x_center) / FOCAL_LEN

[0350] v=DEFOY(y)=(y_center-y) / FOCAL_LEN

[0351] Among them, (x_center, y_center) is the center coordinate of the camera's two-dimensional image frame.

[0352] A1.4 A point in a three-dimensional coordinate system can be represented by a three-dimensional vector v = (x, y, z), which points from the origin of the coordinate system to the position of the point.

[0353] A2 Quaternions, 3D Vectors, 3x3 Matrices and 2D Vector Math

[0354] Quaternions consist of 4 elements

[0355] q=(w,x,y,z)

[0356] A2.1.2 Unit quaternion:

[0357] q=q_idt(q)=(1,0,0,0)

[0358] A2.1.3 Conjugation of quaternions:

[0359] q_cnj(q)=(w,--x,--y,--z)

[0360] A2.1.4 Length of quaternion:

[0361] q_len(q)=sqrt(w*w+x*x+y*y+z*z)

[0362] sqrt() is the square root of a floating point number

[0363] A2.1.5 The length of a unit quaternion is 1

[0364] Normalized quaternion q:

[0365] u=q_uni(q)

[0366] in,

[0367] q=(w,x,y,z)

[0368] u=(uw,ux,uy,uz)

[0369] uw=x / len

[0370] ux=x / len

[0371] uy=y / len

[0372] uz=z / len

[0373] len=q_len(q)

[0374] A2.1.62 The product of quaternions p and q

[0375] t=q_prd2(q,p)=q*p

[0376] in,

[0377] q=(qw,qx,qy,qz)

[0378] p=(pw,px,py,pz)

[0379] t=(tw,tx,ty,tz)

[0380] and,

[0381] tw=(qw*pw-qx*px-qy*py-qz*pz)

[0382] tx=(qw*px+qx*pw+qy*pz-qz*Py)

[0383] ty=(qw*py-qx*pz+qy*pw+qz*px)

[0384] tz=(qw*pz+qx*py-qy*px+qz*pw)

[0385] Because quaternions can be used to represent rotational transformations, if q2=q_prd2(q1,q0) is the product of two quaternions, then using q2 as an orientation transformation is equivalent to first applying q0 and then applying q1. A2.1.7 Product of 3 quaternions:

[0386] q=q_prd3(q1, q2, q3)=q_prd2(q1, q_prd2(q2, q3))

[0387] A2.1.84 The product of quaternions:

[0388] q=q_prd4(q1, q2, q3, q4)=q_prd2(q1, q_prd3(q2, q3, q4))

[0389] A2.1.9 The height difference between two quaternions:

[0390] q=q_ldf(q1, q2)=q_prd2(q_cnj(q1), q2)

[0391] A2.2.1 A three-dimensional vector has three elements:

[0392] v = (x, y, z)

[0393] A2.2.2 Length of a three-dimensional vector:

[0394] v_len(v)=sqrt(x*x+y*y+z*z)

[0395] A2.2.3 The length of a unit three-dimensional vector is 1

[0396] Normalize a 3D vector:

[0397] u=v_uni(v)

[0398] in,

[0399] v = (x, y, z)

[0400] u=(ux,uy,uz)

[0401] ux=x / len

[0402] uy=y / len

[0403] uz=z / len

[0404] len=v_len(v)

[0405] A2.2.4 A unit quaternion can be interpreted as a combination of a rotation vector and an angle around that vector:

[0406] q=(w,x,y,z)

[0407] v = (vx, vy, vz) is the rotation vector

[0408] theta is the rotation angle where,

[0409] w=Cos(theta / 2)

[0410] x=vx*sin(theta / 2)

[0411] y=vy*sin(theta / 2)

[0412] z=vz*sin(theta / 2)

[0413] A2.2.5 The dot product of two three-dimensional vectors va and vb:

[0414] d=v_dot(va, vb)=va.vb=ax*bx+ay*by+az*bz

[0415] in,

[0416] va=(ax,ay,az)

[0417] vb=(bx,by,bz)

[0418] Assume theta is the angle between va and vb:

[0419] Then cos(theta)=v_dot(va,vb)

[0420] A2.2.6 The cross product of two three-dimensional vectors va and vb:

[0421] vc = v_crs(va, vb) = va x vb

[0422] in,

[0423] va=(ax,ay,az)

[0424] vb=(bx,by,bz)

[0425] vc=(cx,cy,cz)

[0426] cx=ay*bz-az*by

[0427] cy=az*bx-ax*bz

[0428] cz=ax*by-ay*bx

[0429] A2.2.7 Rotation vector obtained from quaternion q

[0430] v = v_frm_q(q) = (x, y, z)

[0431] in

[0432] q=(qw,qx,qy,qz)

[0433] len = v_len(qx,qy,qz)

[0434] x=qx / len

[0435] y=qy / len

[0436] z=qz / len.

[0437] A2.2.8 Unit vector of the vector product of two three-dimensional vectors va and vb

[0438] v=v_cru(q)=v_uni(v_crs(va,vb))

[0439] A2.2.9 Rotation angle of the unit quaternion q

[0440] deg = deg_frm_q(q)

[0441] in

[0442] q=(qw,qx,qy,qz)

[0443] deg=2*acos(qw)

[0444] acos() is the arc cosine, the result is in degrees

[0445] A2.2.10 Addition of two three-dimensional vectors va and vb

[0446] v=v_add(va,vb)=(x,y,z)

[0447] va=(ax,ay,az)

[0448] vb=(bx,by,bz)

[0449] x=ax+bx

[0450] y=ay+by

[0451] z=az+bz.

[0452] A2.2.11 The opposite vector of the three-dimensional vector va

[0453] v = v_neg(va) = (x, y, z)

[0454] va=(ax,ay,az)

[0455] x=-ax

[0456] y=-ay

[0457] Z=-aZ.

[0458] A2.3.1 3×3 matrix

[0459]

[0460] A2.3.2 Identity 3×3 matrix

[0461]

[0462] A2.3.3 Matrix subtraction

[0463]

[0464] A2.3.4 Matrix-vector multiplication

[0465] vd = mv_prd (m, v) = m*vs

[0466]

[0467] vs = (x, y, z)

[0468] vd=(dx,dy,dz)

[0469] in,

[0470] dx=Xx*x+Yx*y+Zx*z

[0471] dy=Xy*x+Yy*y+Zy*z

[0472] dz=Xz*x+Yz*y+Zz*z A2.3.5 Quaternion Matrix

[0473] m=m_frm_q(q)

[0474] q=(qw,qx,qy,qz)

[0475] Where m is a 3×3 matrix

[0476]

[0477] and

[0478] Xx=1.0f-2.0f*qy*qy-2.0f*qz*qz Xy=2.0f*qx*qy+2.0f*qw*qz

[0479] Xz=2.0f*qx*qz-2.0f*qw*qy

[0480] Yx=2.0f*qx*qy-2.0f*qw*qz

[0481] Yy=1.0f-2.0f*qx*qx-2.0f*qz*qz Yz=2.0f*qy*qz+2.0f*qw*qx

[0482] Zx=2.0f*qx*qz+2.0f*qw*qy

[0483] Zy=2.0f*qy*qz-2.0f*qw*qx

[0484] Zz=1.0f-2.0f*qx*qx-2.0f*qy*qy A2.3.6 Converting a three-dimensional vector v using quaternion q

[0485] vd=qvq_trans(q,vs)=mv_prd(m,vs)

[0486] q is the quaternion, vs is the original three-dimensional vector

[0487] vd is the transformed three-dimensional vector

[0488] m is a 3×3 matrix

[0489] m=m_frm_q(q)

[0490] A2.3.7 Matrix for rotating the x-axis

[0491] m = m_frm_x_axis_sc(s,c)

[0492] in,

[0493]

[0494] s = sin(theta)

[0495] c=cos(theta)

[0496] and,

[0497] Xx=1.0

[0498] Yx=0.0

[0499] Zx=0.0

[0500] Xy=0.0

[0501] Yy=c

[0502] Zy=-s

[0503] Xz=0.0

[0504] Yz=s

[0505] Zz=c

[0506] A2.3.8 Matrix for rotating the y-axis

[0507] m = m_frm_y_axis_sc(s,c)

[0508] in,

[0509]

[0510] s = sin(theta)

[0511] c=cos(theta)

[0512] and,

[0513] Xx=c

[0514] Yx=0.0

[0515] Zx=s

[0516] Xy=OO

[0517] Yy=1.0

[0518] Zy=0.0

[0519] Xz=-s

[0520] Yz=0.0

[0521] Zz=c

[0522] A2.3.9 Quaternions of matrices

[0523] q=q_frm_m(m)where,

[0524] q=(w,x,y,z)

[0525]

[0526] and,

[0527] if(Zz<0){

[0528] if(Xx>Yy){

[0529] t=1+Xx-Yy-Zz

[0530] w=Yz-Zy

[0531] x=t

[0532] y=Xy+Yx

[0533] z=Zx+Xz

[0534] }else{

[0535] t=1-Xx+Yy-Zz

[0536] w=Zx-Xz

[0537] x=Xy+Yx

[0538] y=t

[0539] z=Yz+Zy

[0540] }

[0541] }else{

[0542] if(Xx<-Yy){

[0543] t=1-Xx-Yy+Zz

[0544] w=Xy-Yx

[0545] x=Zx+Xz

[0546] y=Yz+Zy

[0547] z=t

[0548] }else{

[0549] t=1+Xx+Yy+Zz

[0550] w=t

[0551] x=Yz-Zy

[0552] y=Zx-Xz

[0553] z=Xy-Yx

[0554] }

[0555] }

[0556] A2.3.10 Matrices of vectors

[0557] m = m_frm_v(vx, vy, vz)

[0558] in,

[0559]

[0560] vx=(Xx,xy,Xz)

[0561] vy=(Yx,Yy,Yz)

[0562] vz=(Zx,Zy,Zz)

[0563] A2.4.1 A point in two-dimensional space is a two-dimensional vector, which consists of two elements:

[0564] p=(x,y)

[0565] A 2.4.2 The distance between two-dimensional points pa and pb:

[0566] d=p_dist(pa, pb)=sqrt((xa-xb)*(xa-xb)+(ya-yb)*(ya-yb))where

[0567] pa=(xa,ya)

[0568] pb=(xb,yb)

[0569] A2.4.3 The length of a two-dimensional vector;

[0570] p_len(p)=sqrt(x*x+y*y)

[0571] A2.4.4 The length of a unit two-dimensional vector is 1

[0572] Normalized two-dimensional vector p:

[0573] u=p_uni(p)

[0574] in,

[0575] p=(x,y)

[0576] u=(ux,uy)

[0577] ux=x / len

[0578] uy=y / len

[0579] len=p_len(v)

[0580] A2.4.5 The dot product of two-dimensional vectors pa and pb:

[0581] d=p_dot(pa,pb)=xa*xb+ya*yb

[0582] in,

[0583] pa=(xa,ya)

[0584] pb=(xb,yb)

[0585] The dot product of vectors is of great significance.

[0586] Assume that the angle between vectors pa and pb is theta

[0587] So

[0588] Cos(theta) = p_dot(pa, pb)

[0589] A2.4.6 From 3D vectors to 2D points

[0590] p=p_frm_v(v_in,p_cent,foc_len)

[0591] Where p = (xp, yp) is a point on a two-dimensional plane;

[0592] v = (xv, yv, zv) is a three-dimensional vector in the camera coordinate system CS-C;

[0593] p_cent = (xc, yc) is the center point in the image plane;

[0594] foc_len is the focal length of the camera in pixels

[0595] So we have:

[0596] xp = xv*scale+xc;

[0597] yp=-yv*scale+yc;

[0598] Where scale = foc_len / zv.

[0599] Although the present invention discloses multiple aspects and embodiments, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed in this disclosure are for illustration only and are not intended to be limiting, with the true scope and spirit being described by the appended claims.

Claims

1. A method comprising: Using an imaging sensor, acquiring an image of a first point in the three-dimensional space and an image of a second point in the three-dimensional space; determining a relative orientation of the imaging sensor with respect to a reference system based on the image of the first point and the image of the second point; The first point and the second point are fixed in three-dimensional space.

2. The method according to claim 1, wherein The step of determining the relative orientation of the imaging sensor with respect to the reference system further comprises: determining a vector between a reference point fixed relative to the imaging sensor and the image of the first point, and determining a vector between the reference point and the image of the second point; A relative orientation is determined based on the vector and the vector.

3. The method according to claim 2, in, The vector between the reference point and the image of the first point is a unit vector; The vector between the reference point and the image of the second point is a unit vector.

4. A method comprising: Acquire a first image of a first point in three-dimensional space and a second image of a second point in three-dimensional space using an imaging sensor at its first orientation; acquiring, using an imaging sensor at its second orientation, a third image of the first point and a fourth image of the second point; determining a relative rotation of the imaging sensor between a first orientation and a second orientation based on the first image, the second image, the third image, and the fourth image; The first point and the second point are fixed in three-dimensional space.

5. The method according to claim 4, wherein: The step of determining the relative rotation further comprises: determining, based on the first image and the second image, a relative orientation of the imaging sensor with respect to a reference system when the imaging sensor is in a first orientation; determining, based on the third image and the fourth image, a relative orientation of the imaging sensor with respect to the reference system when the imaging sensor is in the second orientation; A relative rotation of the imaging sensor between the first orientation and the second orientation is determined based on the relative orientation of the imaging sensor with respect to the reference frame when in the first orientation and the relative orientation of the imaging sensor with respect to the reference frame when in the second orientation.

6. The method according to claim 5, wherein: Determining a relative orientation of the imaging sensor with respect to the reference system when the imaging sensor is in the first orientation includes: determining a vector between a reference point fixed relative to the imaging sensor and the first image; and A vector between the reference point and the second image is determined.

7. A method comprising: determining a first relative rotation of the imaging sensor between a first orientation of the imaging sensor and a second orientation of the imaging sensor; determining a second relative rotation of the imaging sensor between a third orientation of the imaging sensor and a fourth orientation of the imaging sensor; Based on the first relative rotation and the second relative rotation, a relative orientation of the imaging sensor with respect to the reference system is determined.

8. The method according to claim 7, wherein The step of determining a first relative rotation comprises: Acquire a first image of a first point in three-dimensional space and a second image of a second point in three-dimensional space using an imaging sensor at its first orientation; acquiring, using an imaging sensor at its second orientation, a third image of the first point and a fourth image of the second point; Based on the first image, the second image, the third image, and the fourth image, a first relative rotation is determined.

9. The method according to claim 7, in, At least three of the first position, the second position, the third position, and the fourth position are different from each other; wherein the first orientation is different from the second orientation; Wherein, the third orientation is different from the fourth orientation.

10. A method comprising: determining a relative orientation of the imaging sensor with respect to the reference system based on a first relative rotation of the imaging sensor between the first orientation of the imaging sensor and the second orientation of the imaging sensor and a second relative rotation of the imaging sensor between the third orientation of the imaging sensor and the fourth orientation of the imaging sensor; Determine a first output, a second output, a third output, and a fourth output corresponding to the spatial orientation sensor when the imaging sensor is in a first orientation, a second orientation, a third orientation, and a fourth orientation, respectively; determining a relative position of the spatial position sensor with respect to a reference system based on the first output, the second output, the third output, and the fourth output; Based on the relative position of the spatial position sensor with respect to the reference system and the relative position of the imaging sensor with respect to the reference system, the relative position of the spatial position sensor with respect to the imaging sensor is determined.

11. The method according to claim 10, wherein: The relative position of the spatial orientation sensor with respect to the imaging sensor is fixed.

12. The method according to claim 10 further comprising determining the first relative rotation by: Acquire a first image of a first point in three-dimensional space and a second image of a second point in three-dimensional space using an imaging sensor at its first orientation; acquiring, using an imaging sensor at its second orientation, a third image of the first point and a fourth image of the second point; Based on the first image, the second image, the third image and the fourth image, the first relative rotation is determined.

13. The method according to claim 12, in, At least three of the first position, the second position, the third position and the fourth position are different from each other; wherein the first orientation is different from the second orientation; Wherein, the third orientation is different from the fourth orientation.

14. A method comprising: Determine a first relative position of the first spatial position sensor relative to the first imaging sensor based on the relative position of the first spatial position sensor relative to the first reference system and the relative position of the first imaging sensor relative to the first reference system; Determine a second relative position of the second spatial position sensor relative to the second imaging sensor based on the relative position of the second spatial position sensor relative to the second reference system and the relative position of the second imaging sensor relative to the second reference system; determining a third relative position of the first spatial position sensor relative to the second spatial position sensor; Based on the first relative orientation, the second relative orientation, and the third relative orientation, a fourth relative orientation of the first imaging sensor with respect to the second imaging sensor is determined.

15. The method according to claim 14, wherein: The first spatial position sensor, the second spatial position sensor, the first imaging sensor, and the second imaging sensor have a fixed relative position with respect to each other.

16. A method comprising: Acquire a first image of a first point in three-dimensional space and a second image of a second point in three-dimensional space using an imaging sensor at its first orientation; acquiring, using an imaging sensor at its second orientation, a third image of the first point and a fourth image of the second point; Acquire a first output and a second output of the spatial orientation sensor corresponding to when the imaging sensor is in a first orientation and a second orientation respectively; A focal length of the imaging sensor is determined based on the first image, the second image, the third image, the fourth image, the first output, and the second output.

17. The method according to claim 16, wherein: The method for determining the focal length includes: Based on the first image, the second image, the third image, and the fourth image, a relative rotation of the imaging sensor between the first orientation and the second orientation is determined.

18. The method according to claim 16, wherein: The imaging sensor and the spatial orientation sensor have a fixed relative orientation with respect to each other.

19. A method comprising: Obtaining a first output, a second output, a third output, and a fourth output corresponding to the spatial orientation sensor when the spatial orientation sensor is in a first orientation, a second orientation, a third orientation, and a fourth orientation, respectively; Based on the first output, the second output, the third output and the fourth output, a relative position of the spatial position sensor with respect to a reference system is determined.

20. The method according to claim 19, in, At least three of the first position, the second position, the third position and the fourth position are different from each other; wherein the first orientation is different from the second orientation; Among them, the third direction is different from the fourth direction.

21. A method comprising: Based on an image acquired using the first imaging sensor, acquiring a first relative position of the eye of the person relative to the first imaging sensor; acquiring a second relative position of the second imaging sensor relative to the first imaging sensor; Based on the first relative orientation and the second relative orientation, a third relative orientation of the eye relative to the second imaging sensor is determined.

22. A non-transitory machine-readable medium having instructions recorded thereon, which, when executed by a computer, implement the method according to any one of claims 1 to 21.

23. A device comprising: a first imaging sensor; a second imaging sensor; A first spatial orientation sensor; A second spatial orientation sensor; wherein the first spatial orientation sensor, the second spatial orientation sensor, the first imaging sensor and the second imaging sensor have fixed relative orientations relative to each other; as well as A computing device comprising a processor and the non-transitory machine-readable medium of claim 22.

Images