Method for determining relative pose between first device and second device

Abstract

The present disclosure provides a method for determining a relative pose between a first device and a second device. According to the specific implementation scheme, first equipment is provided with a first inertial measurement unit and two reference objects, and second equipment is provided with a second inertial measurement unit and an image sensor; determining a first observation gravity direction through a first inertial measurement unit; determining a second observation gravity direction through a second inertial measurement unit; acquiring two pieces of first position information corresponding to the two reference objects under a first coordinate system corresponding to the first equipment; determining two pieces of second position information corresponding to the two reference objects in a second coordinate system corresponding to second equipment based on an image of the first equipment acquired by an image sensor; based on the first observation gravity direction, the second observation gravity direction, the two pieces of first position information and the two pieces of second position information, the relative pose between the first device and the second device is determined.

Description

Technical Field

[0001] The present disclosure relates to the technical field of relative positioning, and in particular, to a method for determining the relative pose between a first device and a second device. Background Art

[0002] In some application scenarios, it is necessary to perform relative positioning on two devices to obtain the relative pose between the two devices, without paying attention to the absolute positioning of each of the two devices. Summary of the Invention

[0003] According to one aspect of the embodiments of the present disclosure, there is provided a method for determining the relative pose between a first device and a second device. The first device is provided with a first inertial measurement unit and two reference objects, and the second device is provided with a second inertial measurement unit and an image sensor. The method includes: determining a first observed gravity direction through the first inertial measurement unit; determining a second observed gravity direction through the second inertial measurement unit; obtaining two first position information of the two reference objects in a first coordinate system corresponding to the first device; determining two second position information of the two reference objects in a second coordinate system corresponding to the second device based on an image of the first device collected by the image sensor; and determining the relative pose between the first device and the second device based on the first observed gravity direction, the second observed gravity direction, the two first position information, and the two second position information.

[0004] According to another aspect of the embodiments of the present disclosure, there is provided a device for determining the relative pose between a first device and a second device. The first device is provided with a first inertial measurement unit and two reference objects, and the second device is provided with a second inertial measurement unit and an image sensor. The device includes: a first determination module for determining a first observed gravity direction through the first inertial measurement unit; a second determination module for determining a second observed gravity direction through the second inertial measurement unit; an acquisition module for obtaining two first position information of the two reference objects in a first coordinate system corresponding to the first device; a third determination module for determining two second position information of the two reference objects in a second coordinate system corresponding to the second device based on an image of the first device collected by the image sensor; and a fourth determination module for determining the relative pose between the first device and the second device based on the first observed gravity direction, the second observed gravity direction, the two first position information, and the two second position information.

[0005] According to still another aspect of the present disclosure, there is provided a computer-readable storage medium storing a computer program for executing the above method for determining the relative pose between a first device and a second device.

[0006] According to another aspect of the present disclosure, there is provided an electronic device, including: a processor; a memory for storing processor-executable instructions; and the processor for reading the executable instructions from the memory and executing the instructions to implement the above method for determining the relative pose between the first device and the second device. Description of the Drawings

[0007] Figure 1 It is a schematic diagram of an application scenario of some exemplary embodiments of the present disclosure.

[0008] Figure 2 It is a schematic flowchart of a method for determining the relative pose between the first device and the second device provided by some exemplary embodiments of the present disclosure.

[0009] Figure 3 It is a schematic diagram of the first coordinate system and the second coordinate system in some exemplary embodiments of the present disclosure.

[0010] Figure 4 It is a schematic flowchart of a method for determining the relative pose between the first device and the second device provided by some other exemplary embodiments of the present disclosure.

[0011] Figure 5-1 It is a schematic flowchart of a method for determining the relative attitude provided by some exemplary embodiments of the present disclosure.

[0012] Figure 5-2 It is a schematic flowchart of a method for determining the relative attitude provided by some other exemplary embodiments of the present disclosure.

[0013] Figure 6-1 It is a schematic diagram of the principle for determining the first unit vector in some exemplary embodiments of the present disclosure.

[0014] Figure 6-2 It is a schematic diagram of the principle for determining the second unit vector in some exemplary embodiments of the present disclosure.

[0015] Figure 7 It is a schematic flowchart of a method for determining the relative translation provided by some exemplary embodiments of the present disclosure.

[0016] Figure 8 It is a schematic diagram of the installation positions of the binocular cameras in some exemplary embodiments of the present disclosure.

[0017] Figure 9 It is a schematic flowchart of a method for performing outlier position screening processing on a candidate position set provided by some exemplary embodiments of the present disclosure.

[0018] Figure 10-1 It is a schematic flowchart of a method for performing outlier position screening processing on a candidate position set provided by some other exemplary embodiments of the present disclosure.

[0019] Figure 10-2 It is a schematic diagram of the principle for performing outlier position screening processing in some exemplary embodiments.

[0020] Figure 11 It is a schematic structural diagram of a device for determining the relative pose between a first device and a second device provided in some exemplary embodiments of the present disclosure.

[0021] Figure 12 It is a schematic structural diagram of a fourth determination module in some exemplary embodiments of the present disclosure.

[0022] Figure 13 It is a schematic structural diagram of a third determination sub-module in some exemplary embodiments of the present disclosure.

[0023] Figure 14 It is a schematic structural diagram of a third determination sub-module in some other exemplary embodiments of the present disclosure.

[0024] Figure 15 It is a schematic structural diagram of a fourth determination sub-module in some exemplary embodiments of the present disclosure.

[0025] Figure 16 It is a schematic structural diagram of a module involved in performing outlier position screening processing on a candidate position set in some exemplary embodiments of the present disclosure.

[0026] Figure 17 It is a structural diagram of an electronic device provided in some exemplary embodiments of the present disclosure. Detailed implementation manners

[0027] To explain the present disclosure, exemplary embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. It should be understood that the present disclosure is not limited by the exemplary embodiments.

[0028] It should be noted that: unless otherwise specifically stated, the relative arrangements of components and steps, numerical expressions and values set forth in these embodiments do not limit the scope of the present disclosure.

[0029] Exemplary method

[0030] In some application scenarios, relative positioning of two devices is required.

[0031] For example, one of the two devices can be a head-mounted display device, and the other can be a movable platform where the head-mounted display device is located. The head-mounted display device can also be referred to as a Head-Mounted Display (HMD) or a headset. The head-mounted display device can be presented in the form of glasses, helmets, etc. The head-mounted display device can include, but is not limited to, Augmented Reality (AR) glasses, Virtual Reality (VR) glasses, etc. The movable platform can include, but is not limited to, vehicles, ships, airplanes, trains, etc. By relatively positioning the head-mounted display device and the movable platform, the relative pose between the head-mounted display device and the movable platform can be obtained. Referring to the relative pose between the head-mounted display device and the movable platform, processing such as rendering, adjustment, warping, and control can be performed on the virtual image displayed by the head-mounted display device to achieve a specific display effect, such as achieving a 6dof (degree of freedom) display effect.

[0032] For another example, one of the two devices can be a drone, and the other can be a movable platform where the drone is to land. By relatively positioning the drone and the movable platform, the relative pose between the drone and the movable platform can be obtained. Referring to the relative pose between the drone and the movable platform, the motion parameters of the drone and the movable platform can be controlled respectively to enable the drone to land on the movable platform smoothly.

[0033] For yet another example, one of the two devices can be a first aircraft flying in the air and in need of refueling, and the other can be a second aircraft flying in the air and used for fuel supply. By relatively positioning the first aircraft and the second aircraft, the relative pose between the first aircraft and the second aircraft can be obtained. Referring to the relative pose between the first aircraft and the second aircraft, the flight parameters of the first aircraft and the second aircraft can be controlled respectively to enable the second aircraft to refuel the first aircraft smoothly in the air.

[0034] To meet the requirement of relatively positioning two devices, embodiments of the present disclosure provide a method for determining the relative pose between a first device and a second device.

[0035] In some alternative embodiments of the present disclosure, the first device can be the movable platform where the head-mounted display device is located, and the second device can be the head-mounted display device. By executing the method provided by the embodiments of the present disclosure, the relative positioning between the head-mounted display device and the movable platform can be directly achieved.

[0036] In some other alternative embodiments of the present disclosure, the first device may be a portable device removably fixed to a movable platform where the head-mounted display device is located, and the second device may be the head-mounted display device. The removable fixing method may include, but is not limited to, screwing, clamping, plugging, etc. By executing the method provided in the embodiments of the present disclosure, relative positioning between the head-mounted display device and the portable device can be achieved. Since the portable device and the movable platform are relatively fixed, based on the relative positioning result between the head-mounted display device and the portable device, relative positioning between the head-mounted display device and the movable platform can be very easily achieved.

[0037] Of course, the types of the first device and the second device are not limited thereto. It is only necessary to ensure that the first device is one of the two devices that need to perform relative positioning, and the second device is the other of the two devices that need to perform relative positioning, and no further enumeration will be made here.

[0038] The first device and the second device may be respectively provided with an Inertial Measurement Unit (IMU). For the sake of easy distinction, the inertial measurement unit provided in the first device may be referred to as the first inertial measurement unit, and the inertial measurement unit provided in the second device may be referred to as the second inertial measurement unit. It can be understood that the inertial measurement unit is a device that uses an accelerometer and a gyroscope to measure the angular velocity and acceleration of an object. In this way, the first inertial measurement unit can be used to measure the angular velocity and acceleration of the first device in the world coordinate system, and the second inertial measurement unit can be used to measure the angular velocity and acceleration of the second device in the world coordinate system. The world coordinate system can be understood as a reference benchmark coordinate system, and the present disclosure does not limit the specific setting of the world coordinate system.

[0039] In addition to being provided with the first inertial measurement unit, the first device may also be provided with two reference objects. Any one of the two reference objects may be a natural feature existing in the first device itself. For example, it may be a corner point, a bump, etc. on the first device. Or, any one of the two reference objects may be a marker additionally added to assist in realizing the relative positioning between the first device and the second device. For example, it may be an object with a special color or pattern, or it may be a Light Emitting Diode (LED) lamp, etc. The two reference objects may be arranged left and right, or may be arranged up and down. The present disclosure does not limit the arrangement manner of the two reference objects. In an optional example, as Figure 1 shown, the second device may be a head-mounted display device worn by the user, and the two reference objects may be the first marker 110 and the second marker 120, and the first marker 110 and the second marker 120 may be arranged up and down.

[0040] In addition to being provided with a second inertial measurement unit, the second device may also be provided with an image sensor. The image sensor may be a sensor for collecting images. When applying this solution, the light incident surface of the image sensor may face the first device so as to collect an image of the first device.

[0041] In the embodiments of the present disclosure, the directions of gravity in the real physical world may be observed respectively through the first inertial measurement unit and the second inertial measurement unit to obtain the observation results corresponding to the first inertial measurement unit and the second inertial measurement unit respectively. In addition, an image of the first device may be collected through the image sensor, and there may be two reference objects in the image of the first device collected by the image sensor. The observation results corresponding to the first inertial measurement unit and the second inertial measurement unit respectively, and the image of the first device collected by the image sensor may be used to assist in determining the relative pose between the first device and the second device. In this way, the relative positioning between the two devices can be effectively achieved.

[0042] As Figure 2 shown, it is a schematic flowchart of a method for determining the relative pose between a first device and a second device provided by some exemplary embodiments of the present disclosure. Figure 2 The method shown may include step 210, step 220, step 230, step 240, and step 250.

[0043] Step 210, determine a first observed gravity direction through the first inertial measurement unit.

[0044] In step 210, the direction of gravity in the real physical world may be observed through the first inertial measurement unit to obtain a first observed gravity direction.

[0045] In some alternative embodiments of the present disclosure, the direction of gravity in the real physical world may be represented as the direction indicated by g in Figure 3 .

[0046] In some alternative embodiments of the present disclosure, when the first device is in a stationary state, the acceleration of the first device may be measured through the first inertial measurement unit. The direction of the measured acceleration may be considered as the observation result of the direction of gravity in the real physical world. Then, the direction of the measured acceleration may be used as the first observed gravity direction.

[0047] It should be noted that when the first device is in a moving state, the direction of gravity in the real physical world may also be observed, but in order to ensure the observation accuracy, the acceleration generated by the first device due to movement cannot be too large.

[0048] Step 220, determine a second observed gravity direction through the second inertial measurement unit.

[0049] In step 220, the second inertial measurement unit can be used to observe the direction of gravity in the real physical world to obtain the second observed gravity direction. For the specific method of obtaining the second observed gravity direction, refer to the relevant introduction to step 210 above and will not be elaborated here.

[0050] Step 230: Obtain two first position information of two reference objects in the first coordinate system corresponding to the first device.

[0051] In some alternative embodiments of the present disclosure, the first coordinate system corresponding to the first device may refer to a three-dimensional coordinate system constructed with the center of mass of the first device or other position points as the origin. In an alternative example, the first coordinate system may be Figure 3 where the origin is O m , and the three coordinate axes are X m , Y m , Z m of the three-dimensional coordinate system.

[0052] In some alternative embodiments of the present disclosure, the two first position information of the two reference objects may both be in coordinate form. If any of the reference objects has a relatively regular shape, the first position information corresponding to the reference object may refer to the coordinates of the geometric center of the reference object in the first coordinate system. Since the two reference objects are both arranged on the first device, the two reference objects can be considered fixed in the first coordinate system, and the two first position information corresponding to the two reference objects can be fixed and known.

[0053] Step 240: Based on the image of the first device collected by the image sensor, determine two second position information of the two reference objects in the second coordinate system corresponding to the second device.

[0054] In some alternative embodiments of the present disclosure, the second coordinate system corresponding to the second device may refer to a three-dimensional coordinate system constructed with the center of mass of the second device or other position points as the origin. In an alternative example, the second coordinate system may be Figure 3 where the origin is O b , and the three coordinate axes are X b , Y b , Z b of the three-dimensional coordinate system.

[0055] In some alternative embodiments of the present disclosure, there may be two reference objects in the image of the first device collected by the image sensor. According to the image of the first device collected by the image sensor, two second position information corresponding to the two reference objects can be determined through algorithms related to computer vision. The two second position information corresponding to the two reference objects may both be in the form of coordinates. If any reference object has a relatively regular shape, the second position information corresponding to the reference object may refer to: the coordinates of the geometric center of the reference object in the second coordinate system.

[0056] Step 250, based on the first observed gravity direction, the second observed gravity direction, the two first position information, and the two second position information, determine the relative pose between the first device and the second device.

[0057] It should be noted that both the first observed gravity direction and the second observed gravity direction are the observed results of observing the gravity direction in the real physical world. Since the first observed gravity direction is observed by the first inertial measurement unit provided in the first device, and the second observed gravity direction is observed by the second inertial measurement unit provided in the second device, the two observed gravity directions can associate the first coordinate system and the second coordinate system. In addition, the two first position information corresponding to the two reference objects can be considered as the expressions of the positions of the two reference objects in the first coordinate system respectively, and the two second position information corresponding to the two reference objects can be considered as the expressions of the positions of the two reference objects in the second coordinate system respectively. Then, the two first position information and the two second position information can also associate the first coordinate system and the second coordinate system. Therefore, referring to the first observed gravity direction, the second observed gravity direction, the two first position information, and the two second position information, the conversion relationship between the first coordinate system and the second coordinate system can be solved to obtain the relative pose between the first device and the second device. The relative pose between the first device and the second device may refer to: the pose of the second device in the first coordinate system. Therefore, by adopting the embodiments of the present disclosure, through the observation of the gravity direction in the real physical world by two inertial measurement units, and then through the introduction of the image sensor and two reference objects, the relative positioning between the two devices can be effectively realized.

[0058] In some alternative embodiments of the present disclosure, the relative pose between the first device and the second device may include a relative attitude and a relative translation. Here, the relative attitude may refer to: the attitude of the second device in the first coordinate system. Here, the relative translation may refer to: the position of the second device in the first coordinate system.

[0059] As Figure 4 shown, it is a schematic flowchart of a method for determining the relative pose between the first device and the second device provided by some other exemplary embodiments of the present disclosure. Figure 4The method shown may include step 410, step 420, step 430, and step 440. Optionally, the combination of step 410 to step 440 may be an alternative implementation of step 250 of the present disclosure.

[0060] Step 410: Based on two pieces of first position information, determine a first vector in a first coordinate system that points from one of two reference objects to the other.

[0061] Step 420: Based on two pieces of second position information, determine a second vector in a second coordinate system that points from one of two reference objects to the other.

[0062] For ease of description, one of the two reference objects may be referred to as the first reference object and the other as the second reference object hereinafter.

[0063] It should be noted that when determining the first vector and the second vector, it is necessary to ensure that: the starting point of the first vector is the same as the starting point of the second vector, and the ending point of the first vector is also the same as the ending point of the second vector. For example, if the first vector is a vector from the first reference object to the second reference object in the first coordinate system, then the second vector needs to be a vector from the first reference object to the second reference object in the second coordinate system. For another example, if the first vector is a vector from the second reference object to the first reference object in the first coordinate system, then the second vector needs to be a vector from the second reference object to the first reference object in the second coordinate system.

[0064] In an alternative example, the first position information corresponding to the first reference object is (x1, y1, z1), the first position information corresponding to the second reference object is (x2, y2, z2), the second position information corresponding to the first reference object is (x1', y1', z1'), and the second position information corresponding to the second reference object is (x2', y2', z2'). If the first vector is a vector from the first reference object to the second reference object in the first coordinate system, then the first vector may be (x2 - x1, y2 - y1, z2 - z1), and the second vector may be (x2' - x1', y2' - y1', z2' - z1').

[0065] Step 430: Based on the first observed gravity direction, the second observed gravity direction, the first vector, and the second vector, determine the relative attitude.

[0066] As Figure 5-1 shown, it is a schematic flow diagram of a method for determining the relative attitude provided by some exemplary embodiments of the present disclosure. Figure 5-1 The method shown may include step 510, step 520, step 530, step 540, and step 550. Optionally, the combination of step 510 to step 550 may be an alternative implementation of step 430 of the present disclosure.

[0067] Step 510, determine the first cross product result of the first observed gravity direction and the first vector.

[0068] In some alternative embodiments of the present disclosure, the first observed gravity direction may be in vector form. Assume the first observed gravity direction is represented as Figure 3 G in m , and the first vector is represented as (AB) m , then the first cross product result can be represented as G m ×(AB) m .

[0069] Step 520, determine the second cross product result of the second observed gravity direction and the second vector.

[0070] In some alternative embodiments of the present disclosure, the second observed gravity direction may be in vector form. Assume the second observed gravity direction is represented as Figure 3 G in b , and the second vector is represented as (AB) b , then the second cross product result can be represented as G b ×(AB) b .

[0071] Step 530, combine the first observed gravity direction, the first vector, and the first cross product result to obtain a first combined matrix.

[0072] Step 540, combine the second observed gravity direction, the second vector, and the second cross product result to obtain a second combined matrix.

[0073] As introduced above, the first observed gravity direction can be represented as G m , the first vector can be represented as (AB) m , and the first cross product result can be represented as G m ×(AB) m , then the first combined matrix can be represented as [G m , (AB) m , G m ×(AB) m .

[0074] As introduced above, the second observed gravity direction can be represented as G b , the second vector can be represented as (AB) b , and the second cross product result can be represented as G b ×(AB) b , then the second combined matrix can be represented as [G b , (AB) b , G b ×(AB)b .

[0075] Of course, when combining the first observed gravity direction, the first vector, and the first cross product result, and when combining the second observed gravity direction, the second vector, and the second cross product result, the combination order is not limited to the above examples. For example, the first combination matrix can also be expressed as [G m ×(AB) m , G m , (AB) m , and correspondingly, the second combination matrix can also be expressed as [G b ×(AB) b , G b , (AB) b .

[0076] Step 550: Determine the relative attitude by using the inverse matrix of the first combination matrix and the second combination matrix.

[0077] Assume that the first combination matrix is expressed as [G m , (AB) m , G m ×(AB) m , the second combination matrix is expressed as [G b , (AB) b , G b ×(AB) b , and the relative attitude is expressed as R bm . Then, the relative attitude can be determined by using the following formula:

[0078] R bm = [G b , (AB) b , G b ×(AB) b [G m , (AB) m , G m ×(AB) m -1

[0079] For ease of understanding, the principle of the formula used to determine the relative attitude R bm in the above paragraph is introduced below.

[0080] G m and G b can be considered as the expressions of the same gravity direction in the first coordinate system and the second coordinate system respectively. (AB) m and (AB) b are both vectors from the reference object A to the reference object B, and can be considered as the expressions in the first coordinate system and the second coordinate system respectively. G m ​×(AB) m and G b ×(AB) b It can also be considered as the expressions of the same meaning in the first coordinate system and the second coordinate system respectively. Then, there can be:

[0081] R bm* G m = G b

[0082] R bm* (AB) m = (AB) b

[0083] R bm* [G m ×(AB) m = G b ×(AB) b

[0084] By arranging the above three equations, the following equations can be obtained:

[0085] R bm* [G m , (AB) m , G m ×(AB) m = [G b , (AB) b , G b ×(AB) b

[0086] By multiplying both sides of the equation in the previous paragraph by [G m , (AB) m , G m ×(AB) m -1 , the formula for determining the relative attitude R bm can be obtained.

[0087] Figure 5-1 In the embodiment shown, by combining the operation of vector cross product, the operation of combining vectors into a matrix, and the operation of matrix inverse matrix and other operation logics, the relative attitude can be determined efficiently and reliably.

[0088] As Figure 5-2 shown, it is a schematic flow chart of a method for determining relative attitude provided by some other exemplary embodiments of the present disclosure. Figure 5-2 The method shown may include step 560, step 570, step 580, step 585, step 590 and step 595. Optionally, the combination of step 550 to step 595 may be an alternative implementation of step 430 of the present disclosure. ​​

[0089] Step 560: Determine a first rotation matrix for rotating the first observed gravity direction to align with the second observed gravity direction.

[0090] In some alternative embodiments of the present disclosure, both the first observed gravity direction and the second observed gravity direction may be in vector form. Assume the first observed gravity direction is represented as G m , and the second observed gravity direction is represented as G b . Then, a vector angle calculation method can be used to calculate the angle θ from G m to G b , determine the rotation axis as G m ×G b , and the rotation matrix with a rotation angle of θ. The determined rotation matrix can be used as the first rotation matrix. The first rotation matrix for rotating G m to align with G b can be understood as follows: If the magnitudes of G m and G b are the same, through the first rotation matrix, G m can be rotated to completely coincide with G b ; if the magnitudes of G m and G b are different, through the first rotation matrix, G m can be rotated to have the same starting point and the same direction as G b . The first rotation matrix can be represented as R(θ), then R(θ) can satisfy:

[0091] G b = R(θ)G m

[0092] Step 570: Determine a first unit vector in the same direction as the first component; wherein, the first component refers to the component of the first vector perpendicular to the first observed gravity direction.

[0093] In an alternative example, as Figure 6-1 shown, the first vector can be represented as (AB) m , the starting point of the first vector can be represented as A m , the ending point of the first vector can be represented as B m , and the first observed gravity direction can be represented as G m . Then, a perpendicular line can be drawn from A m to G m , and the foot of the perpendicular can be represented as D m . Then, the component of the first vector perpendicular to the first observed gravity direction can be represented as (AD) m , that is, the first component can be represented as (AD) m . A m , Bm , D m The plane where the three of them are located can be called plane A m B m D m plane. Using the following formula, the normal vector n of plane A m B m D m can be calculated m :

[0094] n m =(AB) m ×G m / ||(AB) m ×G m ||

[0095] Assume that the first unit vector in the same direction as the first component is represented as d m , and the following formula can be used to calculate d m :

[0096] d m =G m ×n m / ||G m ×n m ||

[0097] Step 580, determine the second unit vector in the same direction as the second component; where the second component refers to: the component of the second vector perpendicular to the second observed gravity direction

[0098] In an optional example, as Figure 6-2 shown, the second vector can be represented as (AB) b , the starting point of the second vector can be represented as A b , the ending point of the second vector can be represented as B b , and the second observed gravity direction can be represented as G b . Then, a perpendicular line can be drawn from A b to G b , and the foot of the perpendicular can be represented as D b , then the second component can be represented as (AD) b . A b , B b , D b The plane where the three of them are located can be called plane A b B b D b plane. Using the following formula, the normal vector n of plane A b B b D b can be calculated b :

[0099] n b=(AB) b ×G b / ||(AB) b ×G b ||

[0100] Assume that the second unit vector in the same direction as the second component is represented as d b , and d can be calculated using the following formula b :[[]]

[0101] d b =G b ×n b / ||G b ×n b ||

[0102] Step 585: Rotate the first unit vector using the first rotation matrix to obtain the third unit vector.

[0103] Assume that the third unit vector is represented as d m ', and d m ' can be calculated using the following formula

[0104] d m ' = R(θ)d m

[0105] Step 590: Determine the second rotation matrix for rotating the third unit vector to align with the second unit vector.

[0106] In some alternative embodiments of the present disclosure, the vector angle calculation method can be used to calculate the angle α from d m ' to d b , and determine that the rotation axis is G b , and the rotation matrix with the rotation angle of α. The determined rotation matrix can be used as the second rotation matrix. The second rotation matrix is used to rotate d m ' to align with d b can be understood as: through the second rotation matrix, d m ' can be rotated to coincide completely with d b . The second rotation matrix can be represented as R(α), then R(α) can satisfy the following formula

[0107] d b = R(α)d m ' = R(α)R(θ)d m

[0108] Step 595: Determine the relative attitude using the first rotation matrix and the second rotation matrix.

[0109] It should be noted that d b and d mIt can be regarded as the expressions of the same vector in the first coordinate system and the second coordinate system respectively. Assume that the relative attitude is represented as R bm , then d b and d m need to satisfy:

[0110] d b = R bm d m

[0111] Based on this, the following formula can be obtained:

[0112] R bm = R(α)R(θ)

[0113] Obviously, the relative attitude can be determined by using the following set of formulas:

[0114] G b = R(θ)G m

[0115] n m = (AB) m × G m / ||(AB) m × G m ||

[0116] d m = G m × n m / ||G m × n m ||

[0117] n b = (AB) b × G b / ||(AB) b × G b ||

[0118] d b = G b × n b / ||G b × n b ||

[0119] d m ' = R(θ)d m

[0120] d b = R(α)d m '

[0121] R bm = R(α)R(θ)

[0122] It should be noted that for a device equipped with an inertial measurement unit, when the device rotates around the gravity direction in the real physical world, the acceleration data measured by the inertial measurement unit will not change, and the rotation angle of the device around the gravity direction in the real physical world cannot be observed. However, when the inclination angle between the device and the gravity direction in the real physical world changes, this change can be observed through the acceleration data. Figure 5-2 In the illustrated embodiment, the relative inclination angles of the first device and the second device with respect to the gravity direction in the real physical world (equivalent to θ in the above text) can be determined first based on the observation results obtained by observing the gravity direction in the real physical world using the first inertial measurement unit and the second inertial measurement unit respectively. Based on the obtained relative inclination angles, in combination with the first vector and the second vector, the rotation angles of the first device and the second device rotating around the gravity direction (equivalent to α in the above text) can be calculated. The relative inclination angles and the rotation angles can be used together to determine the relative pose. In this way, the relative pose can effectively reflect the relative rotation between the first device and the second device, and the reliability of the relative pose can be better ensured.

[0123] The specific method for determining the relative pose is introduced above. Based on the determined relative pose, step 440 can be executed.

[0124] Step 440: Determine the relative translation based on the two first position information, the two second position information, and the relative pose.

[0125] As Figure 7 shown, it is a schematic flowchart of a method for determining relative translation provided by some exemplary embodiments of the present disclosure. Figure 7 The method shown may include step 710, step 720, and step 730. Optionally, the combination of step 710 to step 730 can be used as an alternative implementation of step 440 of the present disclosure.

[0126] Step 710: Based on the two first position information, determine the third position information of a predetermined point on the connection line between the two reference objects in the first coordinate system.

[0127] Step 720: Based on the two second position information, determine the fourth position information of a predetermined point on the connection line between the two reference objects in the second coordinate system.

[0128] In some alternative embodiments of the present disclosure, the predetermined point on the connection line between the two reference objects may refer to: a point whose distance from the first reference object is the product of the total length of the connection line and a preset coefficient. The preset coefficient can be greater than 0 and less than 1. For example, the preset coefficient can be 1 / 4, 1 / 3, 1 / 2, 2 / 3, etc. It can be understood that if the preset coefficient is 1 / 2, the predetermined point can be the midpoint on the connection line.

[0129] It should be noted that given the coordinates of the two endpoints of a line segment, the coordinates at any given point on the line segment can be obtained through simple geometric calculations. Therefore, in step 710, based on the two first position information, the coordinates of a predetermined point in the first coordinate system can be obtained through simple geometric calculations, and this coordinate can be used as the third position information. Similarly, in step 720, based on the two second position information, the coordinates of a predetermined point in the second coordinate system can be obtained through simple geometric calculations, and this coordinate can be used as the fourth position information.

[0130] Step 730, use the third position information, the fourth position information, and the relative pose to determine the relative translation.

[0131] It should be noted that for any point with coordinates P m in the first coordinate system, using the relative pose and the relative translation, this point can be transformed to the second coordinate system to obtain a point with coordinates P b . Assume that the relative pose is represented as R bm , and the relative translation is represented as t bm , then the transformation formula can be:

[0132] P b = R bm* P m + t bm

[0133] Assume that the first position information corresponding to the first reference object is represented as A m , the first position information corresponding to the second reference object is represented as B m , the second position information corresponding to the first reference object is represented as A b , the second position information corresponding to the second reference object is represented as B b , then the third position information can be represented as f(A m , B m ), the fourth position information can be represented as f(A b , B b ), then there can be:

[0134] f(A b , B b ) = R bm* f(A m , B m ) + t bm

[0135] Transforming the formula in the above paragraph, the following formula can be obtained:

[0136] t bm = f(A b , B b ) - Rbm* f(A m ,B m )

[0137] Assume that the predetermined point is the midpoint on the line, then f(A m ,B m ) is the mean value of A m and B m , and f(A b ,B b ) is the mean value of A b and B b . Accordingly, the formula for obtaining the relative translation t bm can be transformed into:

[0138] t bm =(A b +B b ) / 2-R bm* (A m +B m ) / 2

[0139] Adopting the embodiment shown Figure 7 , combined with the operation logic, can efficiently and reliably determine the relative translation.

[0140] In the embodiments of the present disclosure, using two pieces of first position information, the first vector can be efficiently and reliably determined. Using two pieces of second position information, the second vector can be efficiently and reliably determined. The first vector, the second vector, the first observed gravity direction, and the second observed gravity direction can provide very effective references for the determination of the relative attitude. Two pieces of first position information, two pieces of second position information, and the relative attitude can provide very effective references for the determination of the relative translation. In this way, the relative positioning between the two devices can be effectively realized.

[0141] In some alternative embodiments of the present disclosure, the image sensor may include a binocular camera, and the image of the first device collected by the image sensor may include binocular images.

[0142] Such as Figure 8As shown in the figure, the binocular camera may include a first camera 810 and a second camera 820. The binocular images may include a first image collected by the first camera 810 and a second image collected by the second camera 820. A first reference object may exist in both the first image and the second image. According to the first image and the second image, the coordinates of the first reference object in the camera coordinate system corresponding to the first camera or the second camera may be determined by a triangulation method related to binocular vision. Since the binocular camera is disposed on the second device, the conversion relationship between the camera coordinate system corresponding to the first camera and the second coordinate system may be fixed and known, and the conversion relationship between the camera coordinate system corresponding to the second camera and the second coordinate system may also be fixed and known. Then, in the case where the coordinates of the first reference object in the camera coordinate system corresponding to the first camera or the second camera are determined, according to the fixed and known conversion relationship, the coordinates of the first reference object in the second coordinate system may be obtained, that is, the second position information corresponding to the first reference object is obtained. A second reference object may also exist in both the first image and the second image, and the second position information corresponding to the second reference object may be determined in a similar manner, which will not be elaborated herein.

[0143] In this way, by disposing a binocular camera on the second device and combining the application of the binocular vision positioning method, the two second position information corresponding to the two reference objects can be obtained efficiently and reliably.

[0144] As Figure 9 shown, it is a schematic flowchart of a method for performing outlier position screening processing on a candidate position set provided by some exemplary embodiments of the present disclosure. Figure 9 The method shown may include step 910, step 920, and step 930.

[0145] Step 910, determining a candidate position set corresponding to two reference objects from the image of the first device collected by the image sensor.

[0146] In some alternative embodiments of the present disclosure, the image sensor may include a monocular camera.

[0147] In some alternative embodiments of the present disclosure, after collecting the image of the first device by the image sensor, pixel points suspected of being reference objects may be extracted from the collected image to obtain a plurality of pixel points, and the set of the positions of the extracted pixel points may be used as the candidate position set.

[0148] Step 920, obtaining fifth position information of the optical center of the image sensor in the second coordinate system corresponding to the second device.

[0149] In some alternative embodiments of the present disclosure, the fifth position information of the optical center of the image sensor may be in the form of coordinates. Since the image sensor is disposed on the second device, the image sensor can be considered stationary in the second coordinate system, and the fifth position information of the optical center of the image sensor can be fixed and known.

[0150] Step 930: Based on the first observed gravity direction, the second observed gravity direction, and the fifth position information, perform an outlier position screening process on the candidate position set to obtain a candidate position set after the outlier position screening process.

[0151] Embodiments of the present disclosure may involve outlier positions and inlier positions. The inlier position can be understood as: the true position of the reference object in the image. The outlier position is a term opposite to the inlier position. The outlier position can be understood as: other positions in the image except for the true position of the reference object.

[0152] In the case of obtaining a candidate position set after the outlier position screening process, based on the image of the first device collected by the image sensor, determining the two second position information corresponding to the two reference objects in the second coordinate system corresponding to the second device further includes: based on the candidate position set after the outlier position screening process, determining the two second position information corresponding to the two reference objects in the second coordinate system.

[0153] See Figure 10-1 , which is a schematic flowchart of a method for performing an outlier position screening process on a candidate position set provided by some other exemplary embodiments of the present disclosure. Figure 10-1 The method shown may include Step 1010, Step 1020, and Step 1030. Optionally, the combination of Step 1010 to Step 1030 may be an alternative embodiment of Step 930 of the present disclosure.

[0154] Step 1010: Determine a candidate position pair including a first candidate position and a second candidate position from the candidate position set.

[0155] In some alternative embodiments of the present disclosure, two candidate positions may be randomly selected from the candidate position set, and one of the two selected candidate positions is used as the first candidate position, and the other is used as the second candidate position to obtain a candidate position pair including the first candidate position and the second candidate position.

[0156] Step 1020: Based on the first observed gravity direction, the second observed gravity direction, the first vector, and the optical center position represented by the fifth position information, determine whether the candidate position pair satisfies a preset constraint condition to obtain a determination result.

[0157] In some alternative embodiments of the present disclosure, Step 1020 may include:

[0158] In response to the existence of a target vector in the target vector set, obtain a judgment result for characterizing whether the candidate position pair satisfies the preset constraint condition;

[0159] In response to the non-existence of a target vector in the target vector set, obtain a judgment result for characterizing that the candidate position pair does not satisfy the preset constraint condition;

[0160] The target vector set includes: a vector with a starting point located at one of the first ray and the second ray and an end point located at the other of the first ray and the second ray. The first ray refers to: a ray with a starting point at the optical center position and passing through the first candidate position. The second ray refers to: a ray with a starting point at the optical center position and passing through the second candidate position. The target vector refers to: a vector whose included angle with the second observed gravity direction is equal to the target angle. The target angle refers to: the included angle between the first vector and the first observed gravity direction.

[0161] In an optional example, as Figure 10-2 shown, the fifth position information of the optical center of the image sensor can be the coordinates of P1. The first candidate position can be represented as P2, and the second candidate position can be represented as P3. The first ray can be a ray with a starting point at P1 and passing through P2. The second ray can be a ray with a starting point at P1 and passing through P3. The target vector set can include all vectors with a starting point located at one of the first ray and the second ray and an end point located at the other of the first ray and the second ray. The first observed gravity direction can be represented as G m , and the second observed gravity direction can be represented as G b , and the first vector can be represented as (AB) m .

[0162] It should be noted that theoretically, if both the first candidate position and the second candidate position belong to the inlier positions, that is, the first candidate position is the true position of one of the first reference object and the second reference object in the image, and the second candidate position is the true position of the other of the first reference object and the second reference object in the image, then there must be a vector in the target vector set that satisfies the following condition: the included angle with G b is equal to (AB) m and G mThe included angle. Vectors that meet this condition can be considered target vectors. Therefore, a search can be performed in the target vector set to determine whether there is a target vector in the target vector set. If there is a target vector in the target vector set, it can be considered that both the first candidate position and the second candidate position may belong to the inlier positions. Then, it can be determined that the candidate position pair meets the preset constraint condition, and a judgment result indicating that the candidate position pair meets the preset constraint condition can be obtained. If there is no target vector in the target vector set, it can be considered that at least one of the first candidate position and the second candidate position belongs to the outlier position. Then, it can be determined that the candidate position pair does not meet the preset constraint condition, and a judgment result indicating that the candidate position pair does not meet the preset constraint condition can be obtained.

[0163] Step 1030, based on the judgment result, perform an outlier position screening process on the candidate position set.

[0164] If the judgment result indicates that the candidate position pair meets the preset constraint condition, the first candidate position and the second candidate position in the candidate position set can be temporarily retained.

[0165] If the judgment result indicates that the candidate position pair does not meet the preset constraint condition, a third candidate position different from the first candidate position and the second candidate position can be determined from the candidate position set, and it can be determined whether the candidate position pair including the first candidate position and the third candidate position meets the preset constraint condition. If the first candidate position does not meet the preset constraint condition no matter which candidate position it is combined with, it can be determined that the first candidate position belongs to the outlier position. Then, the first candidate position can be deleted from the candidate position set.

[0166] In the above manner, candidate positions that obviously belong to the outlier position can be deleted from the candidate position set to obtain a candidate position set after the outlier position screening process.

[0167] In some examples of the present disclosure, the method of performing an outlier position screening process on the candidate position set can be first performed using a monocular image. After obtaining the candidate position set after the outlier position screening process, the method of determining two second position information corresponding to two reference objects in the second coordinate system can be performed using a binocular image to determine the relative pose between the first device and the second device. This solution can reduce power consumption.

[0168] In some alternative embodiments of the present disclosure, if there are only two candidate positions in the candidate position set after the outlier position screening process, the two candidate positions can be distinguished to clarify which candidate position corresponds to the first reference object and which candidate position corresponds to the second position information. The coordinates of the candidate position corresponding to the first reference object can be used to determine the second position information corresponding to the first reference object. The coordinates of the candidate position corresponding to the second reference object can be used to determine the second position information corresponding to the second reference object. It should be noted that if the only two candidate positions in the candidate position set after the outlier position screening process are the first candidate position and the second candidate position in the foregoing text, the target vector obtained corresponding to the candidate position pair including the first candidate position and the second candidate position may be the same vector as the second vector in the foregoing text. In an alternative example, the only two candidate positions remaining in the candidate position set after the outlier position screening process are Figure 10-2 P2 and P3 in

[0169] and P2 corresponds to the first reference object and P3 corresponds to the second reference object, then the intersection point of the ray starting from P1 and passing through P2 and the target vector can be calculated, and the coordinates of this intersection point can be used as the second position information corresponding to the first reference object. Similarly, the intersection point of the ray starting from P1 and passing through P3 and the target vector can be calculated, and the coordinates of this intersection point can be used as the second position information corresponding to the second reference object.

[0170] Figure 10-1 In the illustrated embodiment, the first observed gravity direction, the second observed gravity direction, and the fifth information position of the optical center can provide very effective reference for the outlier position screening process of the candidate position set, whereby the two second position information corresponding to the two reference objects can be accurately determined.

[0171] In some alternative embodiments of the present disclosure, for a scenario where relative positioning needs to be performed between a head-mounted display device and a movable platform on which the head-mounted display device is located, such as Figure 1For the scene shown, a status quantity can be set. For example, a status quantity including the following parameters can be set: the relative speed, relative translation, relative pose between the head-mounted display device and the movable platform, the angular velocity bias and acceleration bias of the inertial measurement unit disposed on the head-mounted display device, and the angular velocity bias and acceleration bias of the inertial measurement unit disposed on the movable platform. Additionally, a predetermined algorithm can be introduced to estimate and correct the status quantity so as to accurately obtain the relative pose between the head-mounted display device and the movable platform. Based on the obtained relative pose, the virtual image displayed through the head-mounted display device can be processed to achieve a specific display effect. The predetermined algorithm can include, but is not limited to, the Kalman filtering algorithm, the least squares method, etc. Generally speaking, a predetermined algorithm such as the Kalman filtering algorithm needs the initial values of each parameter in the status quantity to run, that is, each parameter in the status quantity needs to be initialized. The method for determining the relative pose between the first device and the second device provided by the embodiments of the present disclosure can be used for the initialization of the relative pose and relative translation in the status quantity.

[0172] In summary, by adopting the embodiments of the present disclosure, the relative positioning between two devices can be effectively achieved.

[0173] Any method for determining the relative pose between the first device and the second device provided by the embodiments of the present disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices, servers, etc. Alternatively, any method for determining the relative pose between the first device and the second device provided by the embodiments of the present disclosure can be executed by a processor. For example, the processor executes any method for determining the relative pose between the first device and the second device mentioned in the embodiments of the present disclosure by calling the corresponding instructions stored in the memory. Details will not be described hereinafter.

[0174] Exemplary device

[0175] The embodiments of the present disclosure provide a device for determining the relative pose between a first device and a second device. Among them, the first device is provided with a first inertial measurement unit and two reference objects, and the second device is provided with a second inertial measurement unit and an image sensor.

[0176] See Figure 11 , which is a schematic structural diagram of a device for determining the relative pose between a first device and a second device provided by some exemplary embodiments of the present disclosure. Figure 11The device shown includes: a first determination module 10 for determining a first observed gravity direction through a first inertial measurement unit; a second determination module 20 for determining a second observed gravity direction through a second inertial measurement unit; an acquisition module 30 for acquiring two first position information of two reference objects in a first coordinate system corresponding to a first device; a third determination module 40 for determining two second position information of two reference objects in a second coordinate system corresponding to a second device based on an image of the first device collected by an image sensor; and a fourth determination module 50 for determining a relative pose between the first device and the second device based on the first observed gravity direction, the second observed gravity direction, the two first position information, and the two second position information.

[0177] In some alternative embodiments of the present disclosure, the relative pose includes: a relative attitude and a relative translation; as Figure 12 shown, the fourth determination module 50 includes: a first determination sub-module 501 for determining a first vector in the first coordinate system pointing from one of the two reference objects to the other based on the two first position information; a second determination sub-module 503 for determining a second vector in the second coordinate system pointing from one of the two reference objects to the other based on the two second position information; a third determination sub-module 505 for determining the relative attitude based on the first observed gravity direction, the second observed gravity direction, the first vector, and the second vector; and a fourth determination sub-module 507 for determining the relative translation based on the two first position information, the two second position information, and the relative attitude.

[0178] In some alternative embodiments of the present disclosure, as Figure 13 shown, the third determination sub-module 505 includes: a first determination unit 5051 for determining a first cross product result of the first observed gravity direction and the first vector; a second determination unit 5053 for determining a second cross product result of the second observed gravity direction and the second vector; a first combination unit 5055 for combining the first observed gravity direction, the first vector, and the first cross product result to obtain a first combination matrix; a second combination unit 5057 for combining the second observed gravity direction, the second vector, and the second cross product result to obtain a second combination matrix; and a third determination unit 5059 for determining the relative attitude using the inverse matrix of the first combination matrix and the second combination matrix.

[0179] In some alternative embodiments of the present disclosure, the relative attitude is determined using the following formula:

[0180] R bm =[G b ,(AB) b ,G b ×(AB) b [G m, (AB) m , G m ×(AB) m -1

[0181] Wherein, R bm represents the relative attitude, G b represents the second observed gravity direction, (AB) b represents the second vector, G m represents the first observed gravity direction, (AB) m represents the first vector.

[0182] In some alternative embodiments of the present disclosure, as Figure 14 shown, the third determination sub-module 505 includes: a fourth determination unit 5061 for determining a first rotation matrix for rotating the first observed gravity direction to align with the second observed gravity direction; a fifth determination unit 5063 for determining a first unit vector in the same direction as the first component; wherein, the first component refers to: the component of the first vector perpendicular to the first observed gravity direction; a sixth determination unit 5065 for determining a second unit vector in the same direction as the second component; wherein, the second component refers to: the component of the second vector perpendicular to the second observed gravity direction; a rotation unit 5067 for rotating the first unit vector by using the first rotation matrix to obtain a third unit vector; a seventh determination unit 5068 for determining a second rotation matrix for rotating the third unit vector to align with the second unit vector; an eighth determination unit 5069 for determining the relative attitude by using the first rotation matrix and the second rotation matrix.

[0183] In some alternative embodiments of the present disclosure, the relative attitude is determined by the following formula:

[0184] G b = R(θ)G m

[0185] n m =(AB) m ×G m / ||(AB) m ×G m ||

[0186] d m = G m ×n m / ||G m ×n m ||

[0187] n b =(AB) b ×G b / ||(AB) b ×G​b ||

[0188] d b = G b × n b / ||G b × n b ||

[0189] d m ' = R(θ)d m

[0190] d b = R(α)d m '

[0191] R bm = R(α)R(θ)

[0192] Wherein, G b represents the second observed gravity direction, G m represents the first observed gravity direction, R(θ) represents the first rotation matrix used to rotate the first observed gravity direction to align with the second observed gravity direction, (AB) m represents the first vector, d m represents the first unit vector in the same direction as the first component, and the first component refers to the component of the first vector perpendicular to the first observed gravity direction, (AB) b represents the second vector, d b represents the second unit vector in the same direction as the second component, and the second component refers to the component of the second vector perpendicular to the second observed gravity direction, d m ' represents the third unit vector obtained by rotating the first unit vector using the first rotation matrix, R(α) represents the second rotation matrix used to rotate the third unit vector to align with the second unit vector, R bm represents the relative attitude.

[0193] In some alternative embodiments of the present disclosure, as Figure 15 shown, the fourth determination sub-module 507 includes: a ninth determination unit 5071 for determining the third position information of a predetermined point on the line connecting two reference objects in the first coordinate system based on two pieces of first position information; a tenth determination unit 5073 for determining the fourth position information of a predetermined point on the line connecting two reference objects in the second coordinate system based on two pieces of second position information; an eleventh determination unit 5075 for determining the relative translation using the third position information, the fourth position information, and the relative attitude.

[0194] In some alternative embodiments of the present disclosure, the relative translation is determined using the following formula:

[0195] t bm = f(Ab , B b ) - R bm* f(A m , B m )

[0196] Wherein, t bm represents relative translation, A b and B b respectively represent one of the second position information in two pieces of second position information, f(A b , B b ) represents the third position information of a predetermined point on the line connecting two reference objects in the first coordinate system determined based on the two pieces of second position information, A m and B m respectively represent one of the first position information in two pieces of first position information, f(A m , B m ) represents the fourth position information of a predetermined point on the line connecting two reference objects in the second coordinate system determined based on the two pieces of first position information.

[0197] In some alternative embodiments of the present disclosure, as Figure 16 shown, the device provided by the embodiments of the present disclosure further includes: a fifth determination module 1610, configured to determine a candidate position set corresponding to two reference objects from the image of the first device collected by the image sensor before determining two pieces of second position information corresponding to the second device in the second coordinate system corresponding to the second device; an acquisition module 1620, configured to acquire the fifth position information of the optical center of the image sensor in the second coordinate system corresponding to the second device; a processing module 1630, configured to perform outlier position screening processing on the candidate position set based on the first observed gravity direction, the second observed gravity direction, and the fifth position information to obtain a candidate position set after outlier position screening processing; a third determination module 40, configured to determine two pieces of second position information corresponding to the two reference objects in the second coordinate system based on the candidate position set after outlier position screening processing.

[0198] In some alternative embodiments of the present disclosure, the image sensor includes: a binocular camera, and the image of the first device collected by the image sensor includes: a binocular image.

[0199] In the device of the present disclosure, the various alternative embodiments, alternative implementation manners, and alternative examples disclosed above can be flexibly selected and combined as needed to achieve corresponding functions and effects, and the present disclosure does not list them one by one.

[0200] Exemplary electronic device

[0201] Figure 17The block diagram of an electronic device according to an embodiment of the present disclosure is illustrated. The electronic device 1700 includes one or more processors 1710 and a memory 1720.

[0202] The processor 1710 can be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and can control other components in the electronic device 1700 to perform desired functions.

[0203] The memory 1720 can include one or more computer program products, and the computer program products can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory can include, for example, random access memory (RAM) and / or cache memory, etc. Non-volatile memory can include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions can be stored on the computer-readable storage medium, and the processor 1710 can run one or more computer program instructions to implement the methods of the various embodiments of the present disclosure described above and / or other desired functions.

[0204] In one example, the electronic device 1700 can further include: an input device 1730 and an output device 1740, and these components are interconnected through a bus system and / or other forms of connection mechanisms (not shown).

[0205] The input device 1730 can further include, for example, a keyboard, a mouse, and so on.

[0206] The output device 1740 can output various information to the outside, which can include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, and so on.

[0207] Of course, for simplicity, Figure 17 only some of the components related to the present disclosure in the electronic device 1700 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application scenarios, the electronic device 1700 can further include any other appropriate components.

[0208] Exemplary computer program product and computer-readable storage medium

[0209] In addition to the above methods and devices, an embodiment of the present disclosure can also be a computer program product, which includes computer program instructions that, when run by a processor, cause the processor to execute the steps in the methods according to various embodiments of the present disclosure described in the "Exemplary Method" section above in this specification.

[0210] A computer program product may write program code for performing the operations of the embodiments of the present disclosure in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and also including conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on a user computing device, partially on the user device, executed as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0211] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, and when the computer program instructions are run by a processor, the processor is caused to execute the steps in the methods according to various embodiments of the present disclosure described in the "Exemplary Methods" section of the present specification above.

[0212] The computer-readable storage medium may adopt any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may, for example, include but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0213] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, the advantages, benefits, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, benefits, effects, etc. are essential for each embodiment of the present disclosure. The specific details disclosed above are only for the purposes of illustration and easy understanding, and not for limitation. The above details do not limit the present disclosure to necessarily adopt the above specific details for implementation.

[0214] Those skilled in the art can make various changes and modifications to the present disclosure without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present disclosure and their equivalent technologies, the present disclosure is also intended to include these changes and modifications.

Claims

1. A method for determining the relative pose between a first device and a second device, wherein the first device is provided with a first inertial measurement unit and two reference objects, and the second device is provided with a second inertial measurement unit and an image sensor. The method includes: Determining a first observed gravity direction through the first inertial measurement unit; Determining a second observed gravity direction through the second inertial measurement unit; Obtaining two first position information corresponding to the two reference objects in a first coordinate system corresponding to the first device; Based on an image of the first device collected by the image sensor, determining two second position information corresponding to the two reference objects in a second coordinate system corresponding to the second device; Determining the relative pose between the first device and the second device based on the first observed gravity direction, the second observed gravity direction, the two first position information, and the two second position information.

2. The method according to claim 1, wherein, The relative pose includes: relative attitude and relative translation; The determining the relative pose between the first device and the second device based on the first observed gravity direction, the second observed gravity direction, the two first position information, and the two second position information includes: Based on the two first position information, determining a first vector in the first coordinate system that points from one of the two reference objects to the other; Based on the two second position information, determining a second vector in the second coordinate system that points from one of the two reference objects to the other; Determining the relative attitude based on the first observed gravity direction, the second observed gravity direction, the first vector, and the second vector; Determining the relative translation based on the two first position information, the two second position information, and the relative attitude.

3. The method according to claim 2, wherein The determining the relative attitude based on the first observed gravity direction, the second observed gravity direction, the first vector, and the second vector includes: Determining a first cross product result of the first observed gravity direction and the first vector; Determining a second cross product result of the second observed gravity direction and the second vector; Combining the first observed gravity direction, the first vector, and the first cross product result to obtain a first combined matrix; Combining the second observed gravity direction, the second vector, and the second cross product result to obtain a second combined matrix; Using the inverse matrix of the first combined matrix and the second combined matrix to determine the relative attitude.

4. The method according to claim 2, wherein The relative attitude is determined using the following formula: R bm = [G b , (AB) b , G b ×(AB) b [G m , (AB) m , G m ×(AB) m -1 ​ Among them, R bm represents the relative attitude, G b represents the second observed gravity direction, (AB) b represents the second vector, G m represents the first observed gravity direction, (AB) m represents the first vector.

5. The method according to claim 2, wherein The determining the relative attitude based on the first observed gravity direction, the second observed gravity direction, the first vector, and the second vector includes: Determining a first rotation matrix for rotating the first observed gravity direction to align with the second observed gravity direction; Determining a first unit vector in the same direction as a first component; wherein the first component refers to the component of the first vector perpendicular to the first observed gravity direction; Determine a second unit vector in the same direction as the second component; wherein, the second component refers to the component of the second vector perpendicular to the second observed gravity direction; Use the first rotation matrix to rotate the first unit vector to obtain a third unit vector; Determine a second rotation matrix for rotating the third unit vector to align with the second unit vector; Use the first rotation matrix and the second rotation matrix to determine the relative pose; 6. The method according to claim 2, wherein The relative pose is determined using the following formula: G b = R(θ)G m n m =(AB) m ×G m / ||(AB) m ×G m || d m = G m × n m / ||G m × n m || n b = (AB) b × G b / ||(AB) b × G b || d b = G b × n b / ||G b × n b || d m ' = R(θ)d m d b = R(α)d m ' R bm = R(α)R(θ) Among them, G b represents the second observed gravity direction, G m represents the first observed gravity direction, R(θ) represents the first rotation matrix for rotating the first observed gravity direction to align with the second observed gravity direction, (AB) m represents the first vector, d m represents the first unit vector in the same direction as the first component, where the first component refers to the component of the first vector perpendicular to the first observed gravity direction, (AB) b represents the second vector, d b represents the second unit vector in the same direction as the second component, where the second component refers to the component of the second vector perpendicular to the second observed gravity direction, d m ' represents the third unit vector obtained by rotating the first unit vector using the first rotation matrix, R(α) represents the second rotation matrix for rotating the third unit vector to align with the second unit vector, R bm represents the relative attitude.

7. The method according to claim 2, wherein The determining of the relative translation based on the two first position information, the two second position information, and the relative pose includes: Based on the two first position information, determine the third position information of a predetermined point on the line connecting the two reference objects in the first coordinate system; Based on the two second position information, determine the fourth position information of a predetermined point on the line connecting the two reference objects in the second coordinate system; Use the third position information, the fourth position information, and the relative pose to determine the relative translation; 8. The method according to claim 2, wherein The relative translation is determined using the following formula: t bm = f(A b , B b ) - R bm* f(A m , B m ) where t bm represents the relative translation, A b and B b respectively represent one of the two second position information, f(A b , B b ) represents the third position information of a predetermined point on the line connecting the two reference objects in the first coordinate system determined based on the two second position information, A m and B m respectively represent one of the two first position information, f(A m , B m ) represents the fourth position information of a predetermined point on the line connecting the two reference objects in the second coordinate system determined based on the two first position information.

9. According to the method of any one of claims 1-8, wherein, Before determining the two second position information of the two reference objects in the second coordinate system corresponding to the second device based on the image of the first device collected by the image sensor, the method further includes: Determine a candidate position set corresponding to the two reference objects from the image of the first device collected by the image sensor; Obtain the fifth position information of the optical center of the image sensor in the second coordinate system corresponding to the second device; Based on the first observed gravity direction, the second observed gravity direction, and the fifth position information, perform outlier position screening processing on the candidate position set to obtain the candidate position set after outlier position screening processing; The determining of the two second position information of the two reference objects in the second coordinate system corresponding to the second device based on the image of the first device collected by the image sensor further includes: Based on the candidate position set after outlier position screening processing, determine the two second position information of the two reference objects in the second coordinate system; 10. According to the method according to any one of claims 1-8, wherein The image sensor includes: a binocular camera, and the image of the first device collected by the image sensor includes: a binocular image; 11. A device for determining the relative pose between a first device and a second device, the first device is provided with a first inertial measurement unit and two reference objects, the second device is provided with a second inertial measurement unit and an image sensor, the device includes: A first determination module, configured to determine a first observed gravity direction through the first inertial measurement unit; A second determination module, configured to determine a second observed gravity direction through the second inertial measurement unit; An acquisition module, configured to acquire two first position information of the two reference objects in the first coordinate system corresponding to the first device; A third determination module, configured to determine two second position information corresponding to the two reference objects in a second coordinate system corresponding to the second device based on an image of the first device acquired by the image sensor; A fourth determination module, configured to determine a relative pose between the first device and the second device based on the first observed gravity direction, the second observed gravity direction, the two first position information, and the two second position information.

12. An electronic device, comprising: A memory, configured to store a computer program product; A processor, configured to execute the computer program product stored in the memory, and when the computer program product is executed, implement the method for determining a relative pose between a first device and a second device according to any one of claims 1-10 above.

13. A computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, implement the method for determining a relative pose between a first device and a second device according to any one of claims 1-10 above.

Images