Method and apparatus for relative positioning of first device and second device

Abstract

Provided are a method for relative positioning of a first device and a second device, and an electronic device and a computer-readable storage medium. The first device (130) is provided with a first inertial measurement unit and a reference object, and the second device (110) is provided with a second inertial measurement unit and an image sensor. The method comprises: acquiring state variables, first data acquired by a first inertial measurement unit, second data acquired by a second inertial measurement unit, and an image having a reference object that is acquired by an image sensor, wherein the state variables comprise: the relative position and relative attitude between a second device and a first device, target parameters, an angular velocity bias and acceleration bias of the first inertial measurement unit, and an angular velocity bias and acceleration bias of the second inertial measurement unit (210); on the basis of the state variables, the first data and the second data, determining prior state variables (220); determining an image feature of the image having the reference object (230); and on the basis of the image feature, correcting the prior state variables to obtain corrected state variables (240).

Description

Method and apparatus for relative positioning of a first device and a second device

[0001] This disclosure claims priority to a Chinese patent application filed with the Patent Office of China on December 29, 2023, with application number CN202311865448.5 and invention name “Method and apparatus for relative positioning of a first device and a second device,” the entire contents of which are incorporated by reference into this disclosure. Technical Field

[0002] The present disclosure relates to the field of relative positioning technology, and in particular to a method and apparatus for relative positioning of a first device and a second device. Background Art

[0003] In some application scenarios, it is necessary to perform relative positioning of two devices without paying attention to the absolute positioning of the two devices. Summary of the Invention

[0004] According to one aspect of the present disclosure, a method for relative positioning of a first device and a second device is provided, wherein the first device is provided with a first inertial measurement unit and a reference object, and the second device is provided with a second inertial measurement unit and an image sensor, the method comprising: acquiring a state quantity, first data collected by the first inertial measurement unit, second data collected by the second inertial measurement unit, and an image with the reference object collected by the image sensor; wherein the state quantity comprises: a relative position and relative posture between the second device and the first device, a target parameter associated with a relative speed between the second device and the first device, an angular velocity bias and an acceleration bias of the first inertial measurement unit, and an angular velocity bias and an acceleration bias of the second inertial measurement unit; determining a priori state quantities based on the state quantities, the first data, and the second data; determining image features of the image with the reference object; and correcting the priori state quantities based on the image features to obtain corrected state quantities.

[0005] According to another aspect of the present disclosure, there is provided an apparatus for relative positioning of a first device and a second device, wherein the first device is provided with a first inertial measurement unit and a reference object, and the second device is provided with a second inertial measurement unit and an image sensor, the apparatus comprising: a first acquisition module for acquiring a state quantity, first data collected by the first inertial measurement unit, second data collected by the second inertial measurement unit, and an image with the reference object collected by the image sensor; wherein the state quantity comprises: the relative position and relative posture between the second device and the first device, the target parameter associated with the relative speed between the second device and the first device, the angular velocity bias and acceleration bias of the first inertial measurement unit, and the angular velocity bias and acceleration bias of the second inertial measurement unit; a first determination module 1 for determining a priori state quantity based on the state quantity, the first data, and the second data; a second determination module for determining image features of the image with the reference object; and a first correction module for correcting the priori state quantity based on the image features to obtain a corrected state quantity.

[0006] According to yet another aspect of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program for executing the above method for relative positioning of a first device and a second device.

[0007] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; and the processor, configured to read the executable instructions from the memory and execute the instructions to implement the above-mentioned method for relative positioning of a first device and a second device. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] FIG1 is a schematic diagram of application scenarios of some exemplary embodiments of the present disclosure.

[0009] FIG2 is a flowchart of a method for relative positioning of a first device and a second device provided by some exemplary embodiments of the present disclosure.

[0010] 3 is a schematic diagram of a lever arm between a first inertial measurement unit and a second inertial measurement unit according to some exemplary embodiments of the present disclosure.

[0011] FIG4 is a flowchart of a method for determining a priori state quantities provided by some exemplary embodiments of the present disclosure.

[0012] FIG5 is a flowchart of a method for determining image features provided by some exemplary embodiments of the present disclosure.

[0013] FIG6 is a flowchart of a method for obtaining a corrected state quantity provided by some exemplary embodiments of the present disclosure.

[0014] FIG7 is a flowchart of a method for obtaining a corrected state quantity provided by other exemplary embodiments of the present disclosure.

[0015] FIG8 is a flowchart of a method for obtaining a corrected state quantity provided by some further exemplary embodiments of the present disclosure.

[0016] FIG9 is a flowchart of a parameter initialization method provided by some exemplary embodiments of the present disclosure.

[0017] FIG10 is a flowchart of a method for determining an initial relative position and an initial relative posture provided by some exemplary embodiments of the present disclosure.

[0018] FIG11 is a flowchart of a method for determining an initialization relative posture provided by some exemplary embodiments of the present disclosure.

[0019] FIG12 is a flowchart of a method for initializing relative posture determination provided by other exemplary embodiments of the present disclosure.

[0020] FIG13-1 is a schematic diagram illustrating a method for determining a first unit vector in some exemplary embodiments of the present disclosure.

[0021] FIG13-2 is a schematic diagram illustrating a method for determining a second unit vector in some exemplary embodiments of the present disclosure.

[0022] FIG14 is a flowchart of a method for handling reference object loss provided by some exemplary embodiments of the present disclosure.

[0023] FIG15-1 is a flowchart of a method for displaying images on a head-mounted display device provided by some exemplary embodiments of the present disclosure.

[0024] FIG15-2 is a flowchart of a method for displaying images on a head-mounted display device provided by some other exemplary embodiments of the present disclosure.

[0025] FIG16 is a flowchart of a reference object retrieval processing method provided by some exemplary embodiments of the present disclosure.

[0026] FIG17 is a schematic structural diagram of an apparatus for relative positioning of a first device and a second device provided by some exemplary embodiments of the present disclosure.

[0027] FIG18 is a schematic structural diagram of a first correction module in some exemplary embodiments of the present disclosure.

[0028] FIG19 is a schematic structural diagram of a first correction module in some other exemplary embodiments of the present disclosure.

[0029] FIG20 is a schematic diagram of modules involved in parameter initialization in some other exemplary embodiments of the present disclosure.

[0030] FIG21 is a schematic diagram of modules involved in reference object loss processing in some exemplary embodiments of the present disclosure.

[0031] Figure 22-1 is a schematic diagram of modules involved in displaying images on a head-mounted display device in some exemplary embodiments of the present disclosure.

[0032] Figure 22-2 is a schematic diagram of modules involved in displaying images on a head-mounted display device in some other exemplary embodiments of the present disclosure.

[0033] FIG. 23 is a schematic diagram of modules involved in a reference object retrieval process in some exemplary embodiments of the present disclosure.

[0034] FIG24 is a structural diagram of an electronic device provided by some exemplary embodiments of the present disclosure. DETAILED DESCRIPTION

[0035] To explain the present disclosure, example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. It should be understood that the present disclosure is not limited to the example embodiments.

[0036] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0037] Exemplary Overview

[0038] In some application scenarios, it is necessary to perform relative positioning of two devices.

[0039] For example, one of the two devices may be a head-mounted display device, and the other may be a movable platform on which the head-mounted display device is located. Also, for example, one of the two devices may be a head-mounted display device, and the other may be an adapter device that is communicatively connected to the head-mounted display device. A head-mounted display device may also be referred to as a head-mounted display (HMD) or a head display. A head-mounted display device may be presented in the form of glasses, a helmet, etc. A head-mounted display device may include but is not limited to augmented reality (AR) glasses, virtual reality (VR) glasses, etc. The movable platform may include but is not limited to a vehicle, a ship, an airplane, a train, etc. By relative positioning the head-mounted display device and the movable platform, or the adapter device, the relative posture between the head-mounted display device and the movable platform, or the adapter device may be obtained. With reference to the relative position between the head-mounted display device and the movable platform, or the adapter device, the virtual image displayed by the head-mounted display device can be rendered, adjusted, distorted (warped), controlled, etc. to achieve a specific display effect, such as a 6dof (degree of freedom) display effect.

[0040] For another example, one of the two devices could be a drone, and the other could be the mobile platform on which the drone is to land. By relative positioning the drone and the mobile platform, the relative pose of the drone and the mobile platform can be determined. Using this relative pose, the motion parameters of the drone and the mobile platform can be controlled to ensure a smooth landing of the drone on the mobile platform.

[0041] For another example, one of the two devices could be a first aircraft in flight that needs to refuel, and the other could be a second aircraft in flight that is being supplied with fuel. By relative positioning the first and second aircraft, the relative position of the first and second aircraft can be determined. Based on this relative position, the flight parameters of the first and second aircraft can be controlled to ensure smooth refueling of the first aircraft by the second aircraft.

[0042] Therefore, how to effectively achieve relative positioning between two devices is a problem worthy of attention for those skilled in the art.

[0043] Exemplary Methods

[0044] In the embodiments of the present disclosure, a Kalman filter algorithm or other optimization algorithms may be used to predict and correct state quantities to effectively achieve relative positioning between two devices.

[0045] An embodiment of the present disclosure provides a method for relative positioning of a first device and a second device.

[0046] In some optional embodiments of the present disclosure, the first device may be a movable platform on which a head-mounted display device is located, and the second device may be the head-mounted display device. By executing the method provided in the embodiments of the present disclosure, relative positioning between the head-mounted display device and the movable platform can be achieved.

[0047] In other optional embodiments of the present disclosure, the first device may be a portable device that is removably fixed to a movable platform on which the head-mounted display device is located, and the second device may be a head-mounted display device. Removable fixing methods may include, but are not limited to, screwing, snapping, plugging, etc. By executing the method provided in the embodiments of the present disclosure, relative positioning between the head-mounted display device and the portable device can be achieved. Since the portable device and the movable platform are relatively fixed, the relative positioning between the head-mounted display device and the movable platform can be easily achieved based on the relative positioning results between the head-mounted display device and the portable device.

[0048] Of course, the types of the first device and the second device are not limited thereto, as long as the first device is one of the two devices that need to be relatively positioned, and the second device is the other of the two devices that need to be relatively positioned.

[0049] The first device and the second device may be respectively provided with an inertial measurement unit (IMU). For ease of distinction, the inertial measurement unit provided on the first device may be referred to as a first inertial measurement unit, and the inertial measurement unit provided on the second device may be referred to as a second inertial measurement unit. It is understandable that an inertial measurement unit is a device that uses an accelerometer and a gyroscope to measure the acceleration and angular velocity of an object. In this way, the first inertial measurement unit can be used to measure the angular velocity and acceleration of the first device in a world coordinate system, and the second inertial measurement unit can be used to measure the angular velocity and acceleration of the second device in a world coordinate system. The world coordinate system can be understood as a reference coordinate system, and the present disclosure does not limit the specific setting of the world coordinate system.

[0050] In addition to setting up the first inertial measurement unit, the first device may also set up a reference object. The reference object may be a natural feature that exists in the first device itself, such as a corner point, a convex point, etc. on the first device. Alternatively, the reference object may be an additional marker (Marker) added to assist in achieving the relative positioning between the second device and the first device, such as an object with a special color or pattern, or a light emitting diode (LED) lamp. The number of reference objects may be two, three, or more than three, which are not listed here one by one.

[0051] In addition to the second inertial measurement unit, the second device may also be provided with an image sensor. The image sensor may be a sensor for capturing images. The image sensor may include at least one camera.

[0052] In an optional example, as shown in FIG1 , the second device may be augmented reality glasses 110 worn by a passenger behind the driver, and the first device may be a portable device 130 removably secured to the driver's seatback 120. A reference object may be provided on the surface of the portable device 130 facing away from the seatback 120. If the rear passenger's head is facing the front of the vehicle, the image sensor provided in the augmented reality glasses 110 may capture an image of the reference object. If the rear passenger's head is facing the left or right of the vehicle, the image sensor provided in the augmented reality glasses 110 may not be able to capture the image of the reference object.

[0053] In some optional embodiments of the present disclosure, the data collected by the first inertial measurement unit, the second inertial measurement unit and the image sensor can be combined to determine the relative position and relative posture between the second device and the first device, so as to effectively achieve relative positioning between the second device and the first device.

[0054] As shown in Figure 2, it is a flowchart of a method for relative positioning of a first device and a second device provided by some exemplary embodiments of the present disclosure. The method shown in Figure 2 may include steps 210, 220, 230, and 240.

[0055] Step 210: Acquire state quantities, first data collected by the first inertial measurement unit, second data collected by the second inertial measurement unit, and an image with a reference object collected by the image sensor; wherein the state quantities include: a relative position and relative posture between the second device and the first device, a target parameter associated with a relative velocity between the second device and the first device, an angular velocity bias and an acceleration bias of the first inertial measurement unit, and an angular velocity bias and an acceleration bias of the second inertial measurement unit.

[0056] It should be noted that both the first device and the second device may have corresponding coordinate systems. The coordinate system corresponding to the first device may be referred to as the first coordinate system. The coordinate system corresponding to the second device may be referred to as the second coordinate system. The first coordinate system may refer to a three-dimensional coordinate system constructed with the center of mass or other position point of the first device as the origin. The second coordinate system may refer to a three-dimensional coordinate system constructed with the center of mass or other position point of the second device as the origin. In an optional example, the first coordinate system may be the coordinate system with the origin O in FIG. m , the three coordinate axes are X m 、Y m 、Z m The second coordinate system can be a three-dimensional coordinate system with the origin O in FIG3 b , the three coordinate axes are X b 、Y b 、Z b 3D coordinate system.

[0057] In some optional embodiments of the present disclosure, the target parameter may include at least one of the following: the relative speed between the second device and the first device; or the result of performing a certain calculation on the relative speed between the second device and the first device. The relative speed between the second device and the first device can be understood as the speed of the second device in the first coordinate system. The relative position and relative posture between the second device and the first device can be understood accordingly.

[0058] In some optional embodiments of the present disclosure, the state quantity obtained in step 210 may be the corrected state quantity of the last correction. Taking the case where the target parameter only includes the relative speed between the second device and the first device as an example, the relative position, relative attitude, and relative speed in the state quantity obtained in step 210 may be the corrected relative position, corrected relative attitude, and corrected relative speed of the last correction, respectively. The angular velocity bias and acceleration bias of the first inertial measurement unit in the state quantity obtained in step 210 may be the corrected angular velocity bias and corrected acceleration bias of the first inertial measurement unit in the last correction. The angular velocity bias and acceleration bias of the second inertial measurement unit in the state quantity obtained in step 210 may be the corrected angular velocity bias and corrected acceleration bias of the second inertial measurement unit in the last correction.

[0059] In some optional embodiments of the present disclosure, the first data acquired in step 210 may include: the most recent angular velocity data and acceleration data acquired by the first inertial measurement unit before the current correction, and historical angular velocity data and historical acceleration data acquired by the first inertial measurement unit before the current correction. The type of the second data acquired in step 210 is similar to the description of the first data and is not further described here.

[0060] It should be noted that, as described above, the number of reference objects can be two, three, or more. The image with the reference objects acquired in step 210 can include all or some of the reference objects, as long as the image with the reference objects can effectively assist in relative positioning between the second device and the first device. In an optional example, the number of reference objects is six, and the number of reference objects included in the image acquired in step 210 can be two, three, six, etc.

[0061] Step 220: Determine a priori state quantity based on the state quantity, the first data, and the second data.

[0062] It should be noted that the a priori state quantity determined in step 220 may be the currently revised a priori state quantity. The currently revised a priori state quantity may include: the a priori parameters corresponding to each parameter in the state quantity obtained in step 210.

[0063] Taking the case where the target parameter only includes the relative speed between the second device and the first device as an example, the first position increment, first attitude increment, and first velocity increment of the first device in the world coordinate system between the last correction and the current correction can be obtained by integrating the first data. Similarly, the second position increment, second attitude increment, and second velocity increment of the second device in the world coordinate system between the last correction and the current correction can be obtained by integrating the second data. Based on the first position increment, first attitude increment, first velocity increment, second position increment, second attitude increment, and second velocity increment, the relative position increment, relative attitude increment, and relative velocity increment between the second device and the first device between the last correction and the current correction can be determined. By combining the relative position increment with the relative position in the state quantity obtained in step 210, the priori parameter corresponding to the relative position can be obtained, which can also be referred to as the priori relative position of the current correction. By combining the relative attitude increment with the relative attitude in the state quantity obtained in step 210, the priori parameter corresponding to the relative attitude can be obtained, which can also be referred to as the priori relative attitude of the current correction. By combining the relative speed increment with the relative speed in the state quantity obtained in step 210, a priori parameters corresponding to the relative speed can be obtained. The priori parameters can also be called the currently corrected priori relative speed.

[0064] The prior relative position, prior relative attitude, and prior relative velocity in the currently corrected prior state are obtained above. Other prior parameters in the prior state can use the corresponding parameters in the state obtained in step 210. For example, the prior angular velocity bias of the first inertial measurement unit in the prior state can be equal to the angular velocity bias of the first inertial measurement unit in the state obtained in step 210. For another example, the prior acceleration bias of the second inertial measurement unit in the prior state can be equal to the acceleration bias of the second inertial measurement unit in the state obtained in step 210.

[0065] It should be noted that the method for determining the prior state quantity is not limited to this. For the sake of clarity, examples will be given later.

[0066] Step 230 : Determine image features of the image with the reference object.

[0067] In some optional embodiments of the present disclosure, the image with the reference object obtained in step 210 may be subjected to feature extraction, matching, and other processing to obtain image features. Image features may include but are not limited to geometric, shape, and color features.

[0068] Step 240: Based on the image features, the prior state quantity is corrected to obtain a corrected state quantity.

[0069] In some optional embodiments of the present disclosure, a certain optimization algorithm can be fixedly adopted to correct the a priori state quantity to obtain a corrected state quantity. Alternatively, a suitable optimization algorithm can be selected according to actual conditions to correct the a priori state quantity to obtain a corrected state quantity. The corrected state quantity obtained here can be the corrected state quantity of the current correction. The corrected state quantity of the current correction can include: the correction parameters corresponding to each a priori parameter in the a priori state quantity determined in step 220. In this way, the corrected state quantity of the current correction can include: the corrected relative position and corrected relative attitude of the current correction. The corrected relative position and corrected relative attitude of the current correction can be considered as the relative positioning result corresponding to the current correction.

[0070] In an embodiment of the present disclosure, a first inertial measurement unit and a reference object can be provided on a first device, and a second inertial measurement unit and an image sensor can be provided on a second device. By combining the data collected by the first inertial measurement unit, the second inertial measurement unit, and the image sensor, state quantities can be predicted and corrected to obtain a corrected relative position and corrected relative attitude between the second device and the first device. In this way, relative positioning between the two devices can be effectively achieved through multi-sensor data fusion.

[0071] In some optional embodiments of the present disclosure, the target parameter may include at least one of the following two: the relative speed between the second device and the first device, and the projection result obtained by projecting the relative speed between the second device and the first device in the inertial coordinate system onto the reference coordinate system, where the reference coordinate system is the coordinate system corresponding to one of the first inertial measurement unit and the second inertial measurement unit. It can be understood that in the case where the target parameter includes the relative speed between the second device and the first device, in step 210, obtaining the state quantity can be understood as obtaining the relative speed between the second device and the first device. In the case where the target parameter includes the projection result obtained by projecting the relative speed between the second device and the first device in the inertial coordinate system onto the reference coordinate system, in step 210, obtaining the state quantity can be understood as obtaining the projection result obtained by projecting the relative speed between the second device and the first device in the inertial coordinate system onto the reference coordinate system.

[0072] In the embodiments of the present disclosure, the first coordinate system can be regarded as the coordinate system corresponding to the first inertial measurement unit, and the second coordinate system can be regarded as the coordinate system corresponding to the second inertial measurement unit. Optionally, the reference coordinate system can be the first coordinate system, and the relative speed between the second device and the first device in the inertial coordinate system can be understood as: the speed difference obtained by subtracting the speed of the second device in the inertial coordinate system from the speed of the first device in the inertial coordinate system. The relative speed between the second device and the first device can be expressed as v, and the projection result obtained by projecting the relative speed between the second device and the first device in the inertial coordinate system to the reference coordinate system can be expressed as v', and v and v' can satisfy the following formula: v'=v+ω1×p

[0073] Wherein, ω1 represents the angular velocity of the first inertial measurement unit, and P represents the relative position between the second device and the first device.

[0074] As shown in Figure 4, it is a flowchart of a method for determining a priori state quantity provided by some exemplary embodiments of the present disclosure. The method shown in Figure 4 may include steps 410 and 420. Optionally, the combination of steps 410 and 420 may be used as an optional implementation of step 220 of the present disclosure.

[0075] Step 410: Obtain a state estimation equation that satisfies a predetermined lever arm constraint condition; wherein the predetermined lever arm constraint condition is defined by at least one of the following first equation and second equation, wherein the first equation defines a relative speed between the second device and the first device, and the second equation defines a relative position between the second device and the first device.

[0076] In some optional embodiments of the present disclosure, the first equation defines a relationship between a derivative of the relative velocity between the second device and the first device and a target difference, where the target difference is the difference obtained by removing the Coriolis acceleration and the drag acceleration from the reference relative acceleration. The reference relative acceleration is associated with the relative attitude between the second device and the first device, the first correction acceleration, and the second correction acceleration. The first correction acceleration is the difference between the acceleration measured by the first inertial measurement unit and the acceleration bias of the first inertial measurement unit. The second correction acceleration is the difference between the acceleration measured by the second inertial measurement unit and the acceleration bias of the second inertial measurement unit. The Coriolis acceleration is associated with the correction angular velocity and the relative velocity between the second device and the first device. The correction angular velocity is the difference between the angular velocity measured by the inertial measurement unit corresponding to the reference coordinate system and the angular velocity bias of the inertial measurement unit. The drag acceleration is associated with the correction angular velocity, the relative position between the second device and the first device, and the derivative of the correction angular velocity.

[0077] Alternatively, if the reference coordinate system is the first coordinate system, the first equation may include:

[0078] in, Indicates the relative speed between the second device and the first device. The symbol · above any parameter indicates a derivative operation. C() indicates converting the parameters in () from quaternion form to rotation matrix form. Indicates the relative posture between the second device and the first device, represents the second corrected acceleration, represents the first corrected acceleration, represents the Coriolis acceleration, represents the corrected angular velocity, and Can form the involved acceleration, Indicates the relative position of the second device to the first device.

[0079] In some optional embodiments of the present disclosure, if the reference coordinate system is the first coordinate system, the second equation may include:

[0080] in, represents the projection result of the relative velocity between the second device and the first device in the inertial coordinate system onto the reference coordinate system. It should be noted that the meanings of the other parameters involved in the second equation can be referred to the relevant explanations of the meanings of the parameters involved in the first equation above, and will not be repeated here.

[0081] It should be noted that the system for achieving relative positioning between the first device and the second device in the embodiment of the present disclosure can be considered as a dual IMU system. The dual IMU system involves two IMUs installed on different devices, namely a first inertial measurement unit provided on the first device and a second inertial measurement unit provided on the second device. The first inertial measurement unit and the second inertial measurement unit cannot overlap in space, and there is a lever arm between the first inertial measurement unit and the second inertial measurement unit. For details, please refer to Figure 3, which will bring about a lever arm effect. The magnitude of the lever arm effect is proportional to and The cross product result is only when The lever arm effect can be ignored only when is 0, or the first inertial measurement unit and the second inertial measurement unit coincide with each other. In view of this, in some embodiments, the first equation can be introduced. It can be understood that, yes The derivative of , then, in the first equation It can reflect the existence of a lever arm between the first inertial measurement unit and the second inertial measurement unit. In other embodiments, a second equation can be introduced. It can be understood that the second equation The lever arm between the first inertial measurement unit and the second inertial measurement unit can be reflected. In this way, it can be considered that the lever arm-related constraints are introduced into the state estimation equation, and the state estimation equation meets the predetermined lever arm constraint conditions.

[0082] Alternatively, the state quantity of the state estimation equation can be expressed as follows:

[0083] Optionally, if the state estimation equation includes the first equation mentioned above, in addition to the first equation, the state estimation equation may further include the following equation:

[0084] Optionally, if the state estimation equation includes the second equation above, in addition to the second equation, the state estimation equation may further include the following equation:

[0085] in, represents the angular velocity bias of the first inertial measurement unit, represents the acceleration bias of the first inertial measurement unit, represents the angular velocity bias of the second inertial measurement unit, Indicates the acceleration bias of the second inertial measurement unit.

[0086] Step 420 : determining a priori state quantity based on the state quantity, the first data, the second data, and the state estimation equation.

[0087] In some optional embodiments of the present disclosure, step 420 may include:

[0088] Obtaining a target analytical solution determined based on a state estimation equation and a preset method; wherein the preset method includes at least one of the following: a Runge-Kutta method and a numerical approximation method; and the target analytical solution refers to an analytical solution of a parameter in a priori state quantities;

[0089] A priori state quantity is determined based on the state quantity, the first data, the second data and the target analytical solution.

[0090] In some optional embodiments of the present disclosure, both the Runge-Kutta method and numerical approximation methods can be used to process angular velocity, acceleration, velocity, translation, and the like. The Runge-Kutta method may include, but is not limited to, the second-order Runge-Kutta method, the third-order Runge-Kutta method, the fourth-order Runge-Kutta method, and the like. It is understood that the Runge-Kutta method is a high-precision, single-step algorithm widely used in engineering. Given known derivatives and initial value information for an equation, computer simulation technology can be used to eliminate the complex process of solving differential equations. Numerical approximation methods may include, but are not limited to, the zero-order approximation method, the first-order approximation method, the second-order approximation method, and the like.

[0091] The zero-order approximation method can be understood as follows: assuming that a parameter needs to be integrated for a period of time, the starting value of the parameter during this period can be used as the zero-order approximation result, or the final value of the parameter during this period can be used as the zero-order approximation result, or the average of the starting value and final value of the parameter during this period can be used as the zero-order approximation result, or the average or weighted average of the starting value, intermediate value, and final value of the parameter during this period can be used as the zero-order approximation result.

[0092] The first-order approximation method can be understood as follows: during the integration process, the angle can be processed by uniform angular acceleration, the acceleration can be processed by uniform acceleration, the velocity can be processed by uniform acceleration, and the translation can be processed by uniform speed.

[0093] The second-order approximation method can be understood as follows: assuming that a parameter needs to be integrated over a period of time, then according to the formula A=a+bt+(1 / 2)ct 2 Perform integration, where a, b, and c represent coefficients, and t represents time. If the parameter is angular velocity, and the start of this period is t0, a can be the angular velocity at t0, b can be the angular acceleration at t0, and c can be the angular jerk at t0. Similarly, if the parameter is velocity, a can be the velocity at t0, b can be the acceleration at t0, and c can be the jerk at t0.

[0094] The various equations included in the state estimation equations described above involve multiple operations, and the different parameters involved in these equations are coupled, making it difficult to directly obtain analytical solutions for the various parameters in the prior state quantities from the state estimation equations. In view of this, in embodiments of the present disclosure, at least one of the Runge-Kutta method and a numerical approximation method can be introduced to simplify the process of solving the analytical solution, thereby obtaining the target analytical solution. In step 420, simply by substituting the state quantity, the first data, and the second data into the target analytical solution, the corresponding prior state quantity can be efficiently and reliably obtained.

[0095] In the embodiment shown in FIG4 , a state estimation equation that conforms to predetermined lever arm constraints can be applied to the process of determining the a priori state. This effectively accounts for the lever arm between the first and second inertial measurement units in the determination of the a priori state. This minimizes the potential errors introduced by the lever arm between the first and second inertial measurement units, allowing for the determination of the true relative velocity and attitude between the second and first devices. This ensures the reliability of the a priori state, and consequently the reliability of the revised state, improving the relative positioning accuracy between the second and first devices.

[0096] FIG5 is a flow chart of a method for determining image features provided by some exemplary embodiments of the present disclosure. The method shown in FIG5 may include steps 510 and 520. The combination of steps 510 and 520 may be an optional implementation of step 230 of the present disclosure.

[0097] Step 510 : Based on the prior relative position and the prior relative posture in the prior state quantity, determine a local area for searching the reference object from the image with the reference object.

[0098] In some optional embodiments of the present disclosure, the optical center position of the image sensor in the second coordinate system can be obtained. Since the image sensor is mounted on the second device, the image sensor can be considered stationary in the second coordinate system, and the optical center position of the image sensor can be fixed and known. Based on the prior relative position and prior relative pose in the prior state variables, the spatial position of the reference object in the second coordinate system can be estimated. If the reference object has a relatively regular shape, the estimated spatial position can refer to the position of the geometric center of the reference object in the second coordinate system. Next, a line connecting the estimated spatial position and the optical center position can be determined, and the intersection of this line with the imaging plane of the image sensor can be considered the estimated pixel position of the reference object. Optionally, based on the estimated pixel position of the reference object, the local area in step 510 can be determined from the image containing the reference object acquired in step 210. For example, a circular area with a preset radius centered at the estimated pixel position of the reference object can be used as the local area in step 510. For another example, a rectangular area with a preset length and width centered at the estimated pixel position of the reference object can be used as the local area in step 510.

[0099] Of course, the method for determining the local area in step 510 is not limited to this. For example, the relative motion pattern between the first and second devices can be determined based on the prior relative position and prior relative posture in the prior state variables. For example, it can be determined whether the first and second devices are relatively close or distant, and in which direction the second device is rotating relative to the first device. The relative motion pattern can be used to estimate the pixel position of the reference object in the image, and based on this, the local area in step 510 can also be determined.

[0100] Step 520: Determine the image features of the local area.

[0101] In some optional embodiments of the present disclosure, feature extraction, matching, and other processing may be performed on the local area to obtain image features.

[0102] In the embodiments of the present disclosure, the prior relative position and prior relative posture in the prior state variables provide a very effective reference for determining a local region. This allows subsequent feature extraction, matching, and other processing to be performed only on the local region, rather than on the entire image, saving computing resources and power consumption.

[0103] FIG6 is a flow chart illustrating a method for obtaining a modified state quantity according to some exemplary embodiments of the present disclosure. The method shown in FIG6 may include steps 610, 620, 630, and 640. Alternatively, a combination of steps 610 to 640 may be used as an optional implementation of step 240 of the present disclosure.

[0104] Step 610: Determine the observed relative posture between the second device and the first device based on the image features.

[0105] In some optional embodiments of the present disclosure, the reference image features can be processed through pixel classification or other processing to determine the actual pixel position of the reference object from the image with the reference object obtained in step 210. Combined with computer vision algorithms, the observed relative pose between the second device and the first device can be determined. Optionally, the observed relative pose can be a 4*4 matrix.

[0106] Step 620: Determine the prior relative position between the second device and the first device based on the prior state quantity.

[0107] It should be noted that the prior state quantity may include: the prior relative position and the prior relative attitude between the second device and the first device. By combining the prior relative position and the prior relative attitude, the prior relative posture can be obtained. For example, the prior relative position can be a vector with a dimension of 3*1, and the prior relative attitude can be a matrix with a dimension of 3*3. By combining the prior relative position and the prior relative attitude, a matrix with a dimension of 4*4 can be obtained as the prior relative posture.

[0108] Step 630: Determine a first residual between the observed relative pose and the prior relative pose.

[0109] In some optional embodiments of the present disclosure, the observed relative pose and the prior relative pose may be subtracted to obtain a first residual. It should be noted that the term "subtraction" in the embodiments of the present disclosure is not limited to simple subtraction, but can be a generalized function that can be used to calculate the difference between different input quantities.

[0110] Step 640: Use the first residual to correct the priori state quantity to obtain a corrected state quantity.

[0111] In some optional embodiments of the present disclosure, the first residual can be used to determine the correction amount corresponding to each prior parameter in the prior state quantity. By combining each determined correction amount with the corresponding prior parameter in the prior state quantity, the correction of all prior parameters in the prior state quantity can be achieved to obtain the corrected state quantity.

[0112] In the embodiment shown in Figure 6, the observed relative posture can be obtained from the image features, and the observed relative posture can be subtracted from the prior relative posture to obtain a first residual for correcting the prior state quantity. This is equivalent to obtaining positioning information based on the inertial measurement unit and the image sensor respectively, and then fusing the positioning information obtained by the two methods to effectively realize the correction of the prior state quantity.

[0113] In some embodiments, the observed relative position and observed relative posture between the second device and the first device can also be determined based on image features. The observed relative position can be subtracted from the prior relative position to obtain a residual. The observed relative posture can be subtracted from the prior relative posture to obtain another residual. Combining these two residuals can also be used to correct the prior state quantity. In this way, the correction of the prior state quantity can be effectively achieved.

[0114] FIG7 is a flow chart of a method for obtaining a modified state quantity provided by another exemplary embodiment of the present disclosure. The method shown in FIG7 may include steps 710 and 720. Optionally, a combination of steps 710 and 720 may be used as an optional implementation of step 240 of the present disclosure.

[0115] Step 710: Determine the observed relative position and observed relative posture between the second device and the first device based on the image features.

[0116] In some optional embodiments of the present disclosure, the reference image features can be used to determine the actual pixel position of the reference object from the image with the reference object obtained in step 210, and then combined with computer vision-related algorithms to determine the observed relative position and observed relative posture between the second device and the first device.

[0117] Step 720 , by fusing the observed relative position, the observed relative posture and the priori state quantity, the priori state quantity is corrected to obtain a corrected state quantity.

[0118] It should be noted that the specific implementation of step 720 can be referred to the relevant introduction in the embodiment shown in Figure 6, which will not be repeated here. In this way, the embodiment shown in Figure 7 can also effectively realize the correction of the prior state quantity.

[0119] FIG8 is a flow chart illustrating a method for obtaining a modified state quantity according to some further exemplary embodiments of the present disclosure. The method shown in FIG8 may include steps 810, 820, 830, 840, and 850. Alternatively, a combination of steps 810 to 850 may be used as an optional implementation of step 240 of the present disclosure.

[0120] Step 810 : Determine a first position of a reference object in an image captured by an image sensor based on the image features.

[0121] Optionally, the method for determining the actual pixel position of the reference object described in the embodiment shown in FIG. 6 may be adopted, and the determined actual pixel position may be used as the first position.

[0122] Step 820: Obtain the optical center position of the image sensor in the coordinate system corresponding to the second device.

[0123] As described above, in the coordinate system corresponding to the second device, the optical center position of the image sensor may be fixed and known.

[0124] Step 830 : Determine a second position of the reference object in the image captured by the image sensor based on the priori relative position and the priori relative posture in the priori state quantity and the position of the optical center.

[0125] Optionally, the method for determining the estimated pixel position of the reference object described in the embodiment shown in FIG. 5 may be adopted, and the determined estimated pixel position may be used as the second position in step 830 .

[0126] Step 840: Determine a second residual between the first position and the second position.

[0127] In some optional embodiments of the present disclosure, the first position and the second position may both be in the form of coordinates, and the second residual may be obtained by subtracting the two coordinates.

[0128] Step 850: Use the second residual to correct the priori state quantity to obtain a corrected state quantity.

[0129] In some optional implementations of the present disclosure, minimizing the second residual may be used as a correction target to correct the priori state quantity to obtain a corrected state quantity.

[0130] In the embodiment shown in FIG8 , a first position can be determined based on image features, a second position can be determined based on the optical center position and the prior state quantity, and the second residual between the first and second positions can be used to correct the predicted state quantity. In other words, there is no need to calculate positioning information based on image features (such as observing relative position and relative posture, or calculating the observed relative posture), but the image features can be used for fusion with the predicted state quantity. In this way, the correction of the prior state quantity can be effectively achieved.

[0131] In the embodiment of the present disclosure, step 240 may include: correcting the priori state quantity by fusing the image feature and the priori state quantity to obtain a corrected state quantity.

[0132] In some optional embodiments of the present disclosure, the number of reference objects may be two. For ease of understanding, hereinafter, one of the two reference objects may be referred to as a first reference object, and the other may be referred to as a second reference object.

[0133] The present disclosure also provides a method for determining the relative posture between a first device and a second device, wherein the first device is provided with a first inertial measurement unit and two reference objects, and the second device is provided with a second inertial measurement unit and an image sensor. The method includes: determining a first observed gravity direction by the first inertial measurement unit; determining a second observed gravity direction by the second inertial measurement unit; obtaining two first position information corresponding to the two reference objects in a first coordinate system corresponding to the first device; determining two second position information corresponding to the two reference objects in a second coordinate system corresponding to the second device based on an image of the first device captured by the image sensor; and determining the relative posture between the first device and the second device based on the first observed gravity direction, the second observed gravity direction, the two first position information, and the two second position information.

[0134] FIG9 is a flow chart of a parameter initialization method provided by some exemplary embodiments of the present disclosure. The method shown in FIG9 may include steps 910, 920, 930, 940, 950, and 960. Steps 910 to 960 may be performed before step 210 of the present disclosure.

[0135] Step 910: Determine a first observed gravity direction using a first inertial measurement unit.

[0136] In step 910 , the direction of gravity in the real physical world may be observed by a first inertial measurement unit to obtain a first observed direction of gravity.

[0137] In some optional embodiments of the present disclosure, the acceleration of the first device can be measured using a first inertial measurement unit when the first device is stationary. The direction of the measured acceleration can be considered as an observation of the direction of gravity in the real physical world. Therefore, the direction of the measured acceleration can be used as the first observed gravity direction.

[0138] It should be noted that when the first device is in motion, the direction of gravity in the real physical world can also be observed. However, in order to ensure the accuracy of the observation, the acceleration of the first device due to the motion cannot be too large.

[0139] Step 920: Determine a second observed gravity direction using a second inertial measurement unit.

[0140] In step 920, the second inertial measurement unit can be used to observe the direction of gravity in the real physical world to obtain a second observed gravity direction. The specific method for obtaining the second observed gravity direction can be referred to the relevant description of step 910 above and will not be repeated here.

[0141] Step 930: Acquire two third positions corresponding to the two reference objects in the coordinate system corresponding to the first device.

[0142] As introduced above, the coordinate system corresponding to the first device may also be referred to as a first coordinate system.

[0143] In some optional embodiments of the present disclosure, the two third positions corresponding to the two reference objects may both be in the form of coordinates. If either reference object has a relatively regular shape, the third position corresponding to the reference object may refer to the coordinates of the geometric center of the reference object in the first coordinate system. Since both reference objects are disposed on the first device, the two reference objects can be considered stationary in the first coordinate system, and the two third positions corresponding to the two reference objects may be fixed and known.

[0144] Step 940: Acquire an initialization image with two reference images captured by the image sensor.

[0145] In some optional embodiments of the present disclosure, the light incident surface of the image sensor may be directed toward the first device, so that an initialization image with two reference objects is captured by the image sensor.

[0146] Step 950 : Determine two fourth positions corresponding to the two reference objects in the coordinate system corresponding to the second device based on the initialization image.

[0147] In some optional embodiments of the present disclosure, based on the initialization image, two fourth positions corresponding to the two reference objects can be determined using computer vision-related algorithms. The two fourth positions corresponding to the two reference objects can both be in the form of coordinates. If either reference object has a relatively regular shape, the fourth position corresponding to the reference object can refer to the coordinates of the geometric center of the reference object in the second coordinate system.

[0148] Step 960 : Determine an initialization relative position and an initialization relative posture between the second device and the first device based on the first observed gravity direction, the second observed gravity direction, the two third positions, and the two fourth positions.

[0149] Initializing the relative position can be used as the initial value of the relative position in the state quantity, and initializing the relative posture can be used as the initial value of the relative posture in the state quantity.

[0150] FIG10 is a flow chart of a method for determining an initial relative position and initial relative posture provided by some exemplary embodiments of the present disclosure. The method shown in FIG10 may include steps 1010, 1020, 1030, and 1040.

[0151] Step 1010 : Determine a first vector pointing from one of the two reference objects to the other in a coordinate system corresponding to the first device based on the two third positions.

[0152] Step 1020 : Determine a second vector pointing from one of the two reference objects to the other in a coordinate system corresponding to the second device based on the two fourth positions.

[0153] It should be noted that when determining the first vector and the second vector, it is necessary to ensure that: the starting point of the first vector and the starting point of the second vector are consistent, and the end point of the first vector and the end point of the second vector are also consistent. For example, if the first vector is a vector pointing from the first reference object to the second reference object in the first coordinate system, then the second vector must be a vector pointing from the first reference object to the second reference object in the second coordinate system. For another example, if the first vector is a vector pointing from the second reference object to the first reference object in the first coordinate system, then the second vector must be a vector pointing from the second reference object to the first reference object in the second coordinate system.

[0154] In an optional example, the first position information corresponding to the first reference object is (x1, y1, z1), the first position information corresponding to the second reference object is (x2, y2, z2), the second position information corresponding to the first reference object is (x1', y1', z1'), and the second position information corresponding to the second reference object is (x2', y2', z2'). If the first vector is a vector pointing from the first reference object to the second reference object in the first coordinate system, the first vector may be (x2-x1, y2-y1, z2-z1), and the second vector may be (x2'-x1', y2'-y1', z2'-z1').

[0155] Step 1030: Determine an initial relative posture based on the first observed gravity direction, the second observed gravity direction, the first vector, and the second vector.

[0156] FIG11 is a flow chart illustrating a method for determining an initial relative posture according to some exemplary embodiments of the present disclosure. The method shown in FIG11 may include steps 1110, 1120, 1130, 1140, and 1150. Alternatively, a combination of steps 1110 through 1150 may be used as an optional implementation of step 1030 of the present disclosure.

[0157] Step 1110 , determining a first cross product result of the first observed gravity direction and the first vector.

[0158] In some optional embodiments of the present disclosure, the first observed gravity direction may be in the form of a vector. Assume that the first observed gravity direction is represented by G m , the first vector is represented as (AB) m, then the first cross product result can be expressed as G m ×(AB) m .

[0159] Step 1120: Determine a second cross product result of the second observed gravity direction and the second vector.

[0160] In some optional embodiments of the present disclosure, the second observed gravity direction may be in the form of a vector. Assume that the second observed gravity direction is represented by G b , the second vector is represented as (AB) b , then the second cross product result can be expressed as G b ×(AB) b .

[0161] Step 1130: Combine the first observed gravity direction, the first vector, and the first cross product result to obtain a first combined matrix.

[0162] Step 1140: Combine the second observed gravity direction, the second vector, and the second cross product result to obtain a second combined matrix.

[0163] As introduced above, the first observed gravity direction can be expressed as G m , the first vector can be expressed as (AB) m , the first cross product result can be expressed as G m ×(AB) m , then the first combination matrix can be expressed as [G m , (AB) m , G m ×(AB) m ].

[0164] As introduced above, the second observed gravity direction can be expressed as G b , the second vector can be expressed as (AB) b , the second cross product result can be expressed as G b ×(AB) b , then the second combination matrix can be expressed as [G b , (AB) b , G b ×(AB) b ].

[0165] Of course, when combining the first observed gravity direction, the first vector, and the first cross product result, and when combining the second observed gravity direction, the second vector, and the second cross product result, the combination order is not limited to the above example. For example, the first combination matrix can also be expressed as [G m ×(AB) m , G m , (AB)m ], accordingly, the second combination matrix can also be expressed as [G b ×(AB) b , G b , (AB) b ].

[0166] Step 1150 : Determine the initial relative posture using the inverse matrix of the first combination matrix and the second combination matrix.

[0167] Assume that the first combination matrix is ​​expressed as [G m , (AB) m , G m ×(AB) m ], the second combination matrix is ​​expressed as [G b , (AB) b , G b ×(AB) b ], the initial relative posture is expressed as R bm , the initial relative posture can be determined using the following formula: R bm =[G b , (AB) b , G b ×(AB) b ][G m , (AB) m , G m ×(AB) m ] -1

[0168] For ease of understanding, the following is the method used to determine the initial relative posture R in the previous paragraph. bm The principle of the formula is introduced.

[0169] G m and G b It can be considered as the expression of the same gravity direction in the first coordinate system and the second coordinate system respectively, (AB) m and (AB) b are all vectors from reference object A to reference object B, which can be considered as expressions in the first coordinate system and the second coordinate system respectively. m ×(AB) m and G b ×(AB) b It can also be considered as the expression of the same meaning in the first coordinate system and the second coordinate system respectively, then we can have: R bm* G m =G b R bm* (AB) m =(AB) b R bm* [Gm ×(AB) m ]=G b ×(AB) b

[0170] By sorting out the above three formulas, we can get the following formula: R bm* [G m , (AB) m , G m ×(AB) m ]=[G b , (AB) b , G b ×(AB) b ]

[0171] By multiplying both sides of the equation in the previous paragraph by [G m , (AB) m , G m ×(AB) m ] -1 , we can get the above-mentioned method for determining the initial relative posture R bm formula.

[0172] In the embodiment shown in FIG11 , various operation logics are combined to efficiently and reliably determine the initialization relative posture.

[0173] FIG12 is a flow chart illustrating a method for initializing relative posture determination according to another exemplary embodiment of the present disclosure. The method illustrated in FIG12 may include steps 1210, 1220, 1230, 1240, 1250, and 1260. Alternatively, a combination of steps 1210 through 1260 may be used as an optional implementation of step 1030 of the present disclosure.

[0174] Step 1210 : Determine a first rotation matrix for rotating the first observed gravity direction to be aligned with the second observed gravity direction.

[0175] In some optional embodiments of the present disclosure, the first observed gravity direction and the second observed gravity direction may both be in the form of vectors. Assume that the first observed gravity direction is represented by G m , the second observed gravity direction is expressed as G b , we can use the vector angle calculation method to calculate the angle between G m Turn G b The included angle θ is determined, and the rotation axis is G m ×G b , the rotation matrix with the rotation angle θ, the determined rotation matrix can be used as the first rotation matrix. The first rotation matrix is ​​used to convert G m Rotate to G bAlignment can be understood as: if G m and G b The modulus length is the same, through the first rotation matrix, G m Rotate to G b Completely overlap; if G m and G b The module lengths are different. Through the first rotation matrix, G m Rotate to G b The starting points coincide and the directions are the same. The first rotation matrix can be expressed as R(θ), then R(θ) can satisfy: G b =R(θ)G m

[0176] Step 1220, determining a first unit vector having the same direction as the first component; wherein the first component refers to: a component of the first vector perpendicular to the first observed gravity direction.

[0177] In an optional example, as shown in FIG13-1, the first vector can be expressed as (AB) m , the starting point of the first vector can be expressed as A m , the end point of the first vector can be expressed as B m , the first observed gravity direction can be expressed as G m Then, we can use A m To G m Draw a perpendicular line, the foot of which can be expressed as D m , then the component of the first vector perpendicular to the first observed gravity direction can be expressed as (AD) m , that is, the first component can be expressed as (AD) m . A m 、B m 、D m The plane where the three are located can be called A m B m D m plane. Using the following formula, A can be calculated m B m D m Normal vector n of the plane m : n m =(AB) m ×G m / ||(AB) m ×G m ||

[0178] Assume that the first unit vector with the same direction as the first component is denoted by d m , d can be calculated using the following formula m : d m =G m ×n m / ||G m ×n m ||

[0179] Step 1230, determining a second unit vector having the same direction as the second component; wherein the second component refers to: a component of the second vector perpendicular to the second observed gravity direction.

[0180] In an alternative example, as shown in FIG13-2, the second vector can be expressed as (AB) b , the starting point of the second vector can be expressed as A b , the end point of the second vector can be expressed as B b , the second observed gravity direction can be expressed as G b Then, we can use A b To G b Draw a perpendicular line, the foot of which can be expressed as D b , then the second component can be expressed as (AD) b . A b 、B b 、D b The plane where the three are located can be called A b B b D b plane. Using the following formula, A can be calculated b B b D b Normal vector n of the plane b : n b =(AB) b ×G b / ||(AB) b ×G b ||

[0181] Assume that the second unit vector with the same direction as the second component is denoted by d b , d can be calculated using the following formula b : d b =G b ×n b / ||G b ×n b ||

[0182] Step 1240: Use the first rotation matrix to rotate the first unit vector to obtain a third unit vector.

[0183] Assume that the third unit vector is denoted as d m ', d can be calculated using the following formula m ': d m ' = R(θ)d m

[0184] Step 1250 : Determine a second rotation matrix for rotating the third unit vector to align with the second unit vector.

[0185] In some optional embodiments of the present disclosure, a vector angle calculation method can be used to calculate the angle between d and d. m 'Turn to d b The included angle α is determined by the rotation axis G b , the rotation matrix with a rotation angle of α, the determined rotation matrix can be used as the second rotation matrix. The second rotation matrix is ​​used to convert d m 'Rotate to the same position as d b Alignment can be understood as: through the second rotation matrix, d m 'Rotate to the same position as d b The second rotation matrix can be expressed as R(α), then R(α) can satisfy the following formula: b =R(α)d m ' = R(α)R(θ)d m

[0186] Step 1260: Determine the initial relative posture using the first rotation matrix and the second rotation matrix.

[0187] It should be noted that d b and d m It can be considered as the expression of the same vector in the first coordinate system and the second coordinate system. Assume that the relative posture is expressed as R bm , then d b and d m Need to meet: d b =R bm d m

[0188] On this basis, the following formula can be obtained: bm =R(α)R(θ)

[0189] Obviously, the following set of formulas can be used to determine the initial relative posture: G b =R(θ)G m n m =(AB) m ×G m / ||(AB) m ×G m || d m =G m ×n m / ||G m ×n m || n b =(AB) b ×G b / ||(AB)b ×G b || d b =G b ×n b / ||G b ×n b || d m ' = R(θ)d m d b =R(α)d m ' R bm =R(α)R(θ)

[0190] In the embodiment shown in Figure 12, the relative inclination angle of the first device and the second device relative to the direction of gravity in the real physical world (equivalent to θ in the above text) can be determined based on the observation results obtained by the first inertial measurement unit and the second inertial measurement unit respectively observing the direction of gravity in the real physical world. On the basis of the relative inclination angle obtained, combined with the first vector and the second vector, the rotation angle of the first device and the second device around the direction of gravity (equivalent to α in the above text) can be calculated. The relative inclination angle and the rotation angle can be used together to initialize the determination of the relative posture. In this way, the initialized relative posture can effectively reflect the relative rotation between the first device and the second device, and the reliability of the initialized relative posture can be better guaranteed.

[0191] For example, an embodiment of the present disclosure provides a method for initializing the relative posture between a first device and a second device, wherein the first device is provided with a first inertial measurement unit and two reference objects, and the second device is provided with a second inertial measurement unit and an image sensor. The method includes: observing the direction of gravity in the real physical world based on the first inertial measurement unit and the second inertial measurement unit, respectively, to obtain observation results; determining the relative inclination angle of the first device and the second device relative to the direction of gravity in the real physical world; and calculating the rotation angle of the first device and the second device around the direction of gravity based on the relative inclination angle, the first vector, and the second vector.

[0192] The specific method for determining the initialization relative posture is described above. Based on the determined initialization relative posture, step 1040 may be executed.

[0193] Step 1040 : Determine an initial relative position based on the two third positions, the two fourth positions, and the initial relative posture.

[0194] It should be noted that for any coordinate in the first coordinate system, P m By using the initialization relative posture and initialization relative position, this point can be transformed into the second coordinate system to obtain the coordinates P b Assume that the initial relative posture is represented by R bm , initialize the relative position to be tbm , then the transformation formula can be: P b =R bm* P m +t bm

[0195] Assume that the third position corresponding to the first reference object is represented by A m , the third position corresponding to the second reference object is represented by B m , the fourth position corresponding to the first reference object is represented by A b , the fourth position corresponding to the second reference object is represented by B b , the following formula can be used to calculate the initial relative position t bm : t bm =(A b +B b ) / 2-R bm* (A m +B m ) / 2

[0196] In this way, through the application of various computational logics, the initial relative posture and initial relative position can be efficiently and reliably obtained. The initial relative posture can be used as the initial value of the relative posture in the state quantity, and the initial relative position can be used as the initial value of the relative position in the state quantity, thereby achieving the initialization of the entire state quantity. After the state quantity is initialized, the initial values ​​can be used to run the optimization algorithm to effectively achieve relative positioning between the second device and the first device.

[0197] It should be noted that, during the actual operation phase of the optimization algorithm, the above initialization scheme can also be referred to for calculating the relative position and relative posture.

[0198] FIG14 is a flow chart of a method for handling reference object loss provided by some exemplary embodiments of the present disclosure. The method shown in FIG14 may include steps 1410 , 1420 , and 1430 .

[0199] Step 1410: Acquire an image captured by an image sensor.

[0200] It should be noted that the image acquired in step 1410 may be any image captured by an image sensor, and the image may contain all reference objects, may not contain any reference objects, or may contain some reference objects.

[0201] Step 1420 , in response to the image captured by the image sensor satisfying a preset loss condition associated with the reference object, determining a target relative position; wherein the target relative position refers to: a corrected relative position in a most recently corrected state quantity obtained before the image captured by the image sensor satisfies the preset loss condition.

[0202] In some optional embodiments of the present disclosure, whether a preset loss condition for reference object association is satisfied may be determined based on the number of reference objects in an image captured by an image sensor. For example, if the number of reference objects in an image captured by the image sensor does not reach a preset number, the preset loss condition for reference object association may be determined to be satisfied. If the number of reference objects in an image captured by the image sensor reaches a preset number, the preset loss condition for reference object association may be determined to be unsatisfied.

[0203] Optionally, the target relative position may be a preset relative position, and the preset relative position may be a fixed value.

[0204] Optionally, when determining whether the preset loss condition for reference object association is met, a time factor may also be considered. For example, if the number of reference objects in images captured by the image sensor does not reach a preset number within a period of time, it may be determined that the preset loss condition for reference object association is met.

[0205] In some optional embodiments of the present disclosure, the state quantity can be maintained, and the prior state quantity and the corrected state quantity can be obtained by predicting and correcting the state quantity. The prior state quantity and the corrected state quantity can be obtained in the manner described above. Of course, the prior state quantity and the corrected state quantity can also be obtained in other ways besides the manner described above. For example, if the lever arm effect caused by the lever arm between the first inertial measurement unit and the second inertial measurement unit is not obvious, when determining the prior state quantity, the state estimation equation that meets the predetermined lever arm constraint condition described above can be used instead of the conventional state estimation equation. For another example, the first inertial measurement unit can be provided with a first magnetometer, and the second inertial measurement unit can be provided with a second magnetometer. The magnetic data collected by the first magnetometer and the second magnetometer can be used as observation data for correcting the prior state quantity, so that the corrected state quantity can also be obtained.

[0206] If the image captured by the image sensor meets the preset loss condition associated with the reference object, the corrected relative position can be obtained from the most recently corrected state quantity obtained before the preset loss condition was met, and the obtained corrected relative position can be used as the target relative position. In an optional example, if the image captured by the image sensor is detected to meet the preset loss condition associated with the reference object at time k, but the preset loss condition has not been met at time k-1, then the corrected relative position in the corrected state quantity at time k can be used as the target relative position.

[0207] Step 1430: Use the target relative position to correct the prior state quantity to obtain a corrected state quantity.

[0208] In some optional embodiments of the present disclosure, a third residual between the target relative position and the a priori relative position in the a priori state quantity can be determined, and the a priori state quantity can be corrected using the third residual to obtain a corrected state quantity. The method for correcting the a priori state quantity using the third residual can be referred to the above description of the method for correcting the a priori state quantity using the first residual, and will not be repeated here.

[0209] In an embodiment of the present disclosure, if the image captured by the image sensor meets the preset loss condition associated with the reference object, it is difficult for the image captured by the image sensor to assist in the relative positioning between the first device and the second device. Then, the corrected relative position in the most recent corrected state quantity obtained before the image captured by the image sensor meets the preset loss condition can be determined, and the corrected relative position can be used as the target relative position for correction of the state quantity. It can be understood that in general usage scenarios, the relative position between the second device and the first device will not change significantly in a short period of time. Accordingly, using the target relative position to correct the prior state quantity has little effect on the accuracy of the corrected relative position and the corrected relative posture in the obtained corrected state quantity. In this way, the relative positioning accuracy between the second device and the first device can be guaranteed as much as possible when the reference object is lost and the image data cannot be used.

[0210] It is understood that the method of the present disclosure can obtain a corrected state quantity, which may include a corrected relative position and relative posture between the second device and the first device. In an embodiment where a head-mounted display device serves as the second device, a rendering engine used to render images of the head-mounted display device can use the corrected state quantity to process the images.

[0211] In some optional embodiments of the present disclosure, the second device may include a head-mounted display device. As shown in Figure 15-1, the method provided by the embodiment of the present disclosure may further include step 1550. Optionally, step 1550 may be performed after step 240 of the present disclosure.

[0212] For example, in step 1550 shown in FIG. 15-1 , the image to be displayed of the head-mounted display device may be rendered based on the corrected relative position and corrected relative posture in the currently obtained corrected state quantity, so that the head-mounted display device can display the image to be displayed.

[0213] It can be understood that displaying the image to be displayed by the head-mounted display device may not be a step of the method in this embodiment, but other additional steps.

[0214] In some optional embodiments of the present disclosure, the rendering engine may render the virtual image with reference to the corrected relative position and corrected relative posture in the currently obtained corrected state quantity, and update the rendering result to the display screen of the head-mounted display device for display.

[0215] In other optional embodiments of the present disclosure, the rendering engine may refer to the corrected relative position and corrected relative posture in the currently obtained corrected state quantity to distort, re-render, adjust, etc. the virtual image to adjust at least one of the display position and posture of the virtual image.

[0216] In some further optional embodiments of the present disclosure, the rendering engine may render the virtual image and configure the optical machine in the head-mounted display device with reference to the corrected relative position and corrected relative posture in the currently obtained corrected state quantity.

[0217] In embodiments of the present disclosure, the corrected relative position and relative posture in the currently obtained corrected state can be used for rendering images on a head-mounted display device. For example, the relative position and relative posture used for rendering images are not fixed, but can change as the first device and the second device undergo relative translation and rotation. In this way, the head-mounted display device can present a 6Dof display effect.

[0218] In some optional embodiments of the present disclosure, the second device may include a head-mounted display device. As shown in FIG15-2 , the method provided in the embodiment of the present disclosure may further include step 1560. Optionally, step 1560 may be performed after step 1430 of the present disclosure.

[0219] For example, in step 1560 shown in FIG. 15-2 , the image to be displayed of the head-mounted display device may be rendered based on the target relative position and the corrected relative posture in the corrected state quantity currently obtained, so that the head-mounted display device can display the image to be displayed.

[0220] It should be noted that the specific implementation of step 1560 can refer to the relevant introduction to step 1550 above, and will not be repeated here.

[0221] In the embodiments of the present disclosure, the target relative position and the corrected relative posture in the current corrected state can be used for image rendering. Then, during the period when the preset loss condition is met, the relative position used for image rendering is fixed to the target relative position, while the relative posture used for image rendering is not fixed. In this way, the head-mounted display device can present a 3D display effect.

[0222] As shown in Figure 16, it is a flowchart of a reference object retrieval processing method provided by some exemplary embodiments of the present disclosure. The method shown in Figure 16 may include steps 1610 and 1620. Optionally, step 1610 may be an optional implementation of step 240 of the present disclosure.

[0223] Step 1610 : In response to the image with the reference object meeting a preset loss and recovery condition associated with the reference object, the prior state quantity is modified based on the image features using an Iterated Extended Kalman Filter (IEKF) algorithm.

[0224] In some optional embodiments of the present disclosure, after a preset loss condition is satisfied, it may be determined whether a preset loss and recovery condition associated with a reference object is satisfied. For example, after the preset loss condition is satisfied, it may be determined whether the number of reference objects in the next image captured by the image sensor reaches a preset number. If, at a certain moment, the number of reference objects in an image captured by the image sensor reaches the preset number, the image may be considered an image containing the reference object and satisfying the preset loss and recovery condition.

[0225] Optionally, a time factor may be considered when determining whether a preset loss and recovery condition associated with a reference object is satisfied. For example, if the number of reference objects in images captured by the image sensor reaches a preset number within a period of time after the preset loss condition is satisfied, then the preset loss and recovery condition associated with the reference object may be determined to be satisfied.

[0226] Step 1620 : In response to the operation information of the iterative extended Kalman filter algorithm satisfying the preset operation condition, the algorithm used for correcting the prior state quantity is switched from the iterative extended Kalman filter algorithm to a predetermined algorithm.

[0227] In some optional embodiments of the present disclosure, the operation information of the iterative extended Kalman filter algorithm may include, but is not limited to, the runtime and number of runs of the iterative extended Kalman filter algorithm. If the runtime is greater than a preset duration, the operation information may be determined to satisfy a preset operation condition. Alternatively, if the number of runs is greater than a preset number, the operation information may be determined to satisfy a preset operation condition.

[0228] In some optional embodiments of the present disclosure, the predetermined algorithm may be the algorithm defined in the method related to FIG. 2 discussed above, the predetermined algorithm may also be an algorithm conventionally used to predict the correction of the state quantity, or other algorithms. The predetermined algorithm may include an extended Kalman filter (EKF) or other optimization algorithms that consume less power than the iterative extended Kalman filter.

[0229] In an embodiment of the present disclosure, if the preset loss and retrieval conditions associated with the reference object are met, the iterative extended Kalman filter algorithm can be first used to correct the prior state quantity. This can reduce the linearization error through multiple iterations to enable rapid data convergence. After the operating information of the iterative extended Kalman filter algorithm meets the preset operating conditions, the iterative extended Kalman filter algorithm can be switched to a predetermined algorithm, such as the algorithm defined in the method related to FIG. 2, so as to minimize the amount of calculation and save power consumption while ensuring the relative positioning accuracy between the second device and the first device.

[0230] In some optional embodiments of the present disclosure, the parameter initialization method described above can be used to first determine the initial relative position and initial relative posture, and based on this, the entire state quantity can be initialized to begin running the optimization algorithm. During the algorithm execution, a priori state quantity can be determined based on the state quantity, first data collected by the first inertial measurement unit, and second data collected by the second inertial measurement unit. The process of determining the predicted state quantity can take into account the lever arm between the first inertial measurement unit and the second inertial measurement unit. In addition, an image sensor provided on the second device can be used to capture an image. If the captured image contains a reference object, the priori state quantity can be corrected based on image features, and a 6Dof effect can be achieved based on this. It should be noted that when using image features, a reference object can be searched for in a local area of ​​the image, that is, a local search can be performed on the image to find the actual pixel position of the reference object. If the local search does not find the reference object, a global search can be performed on the image. Of course, a global search can also be performed directly without performing a local search. If the reference object is lost in the captured image, for example, if a preset loss condition is met, the state quantity can be corrected using the target relative position, and a 3Dof effect can be achieved based on this. If the reference object in the acquired image is lost and then recovered, for example, if the preset loss and recovery conditions are met, the IEKF can be run for a period of time to accelerate convergence before running the EFK.

[0231] In summary, the embodiments of the present disclosure can effectively achieve relative positioning between two devices.

[0232] Optionally, the present disclosure provides a method for achieving relative positioning of a first device and a second device when an image sensor of the second device cannot capture valid image data that can be used for positioning.

[0233] A method for relative positioning of a first device and a second device, wherein the first device is provided with a first inertial measurement unit and a reference object, and the second device is provided with a second inertial measurement unit and an image sensor. The method comprises: acquiring a state quantity, first data collected by the first inertial measurement unit, second data collected by the second inertial measurement unit, and an image collected by the image sensor; wherein the state quantity comprises: a relative position and relative attitude between the second device and the first device, a target parameter associated with the relative velocity between the second device and the first device, an angular velocity bias and an acceleration bias of the first inertial measurement unit, and an angular velocity bias and an acceleration bias of the second inertial measurement unit; determining a priori state quantity based on the state quantity, the first data, and the second data; determining a target relative position in response to an image collected by the image sensor satisfying a preset loss condition associated with the reference object; wherein the target relative position is one of the following: a corrected relative position in the most recently corrected state quantity obtained before the image collected by the image sensor satisfied the preset loss condition, and a preset relative position; and correcting the priori state quantity using the target relative position to obtain a corrected state quantity.

[0234] Optionally, when the target relative position is the corrected relative position in the most recent corrected state quantity obtained before the image captured by the image sensor meets the preset loss condition, the target relative position can be the corrected relative position obtained by the algorithm related to Figure 2, or the corrected relative position obtained by other algorithms.

[0235] Optionally, when the target relative position is a preset relative position, the preset relative position may be a fixed value, which may be a value set in advance according to an application scenario.

[0236] For example, when the first device is a movable device that is in communication with the second device and the user controls the second device using the movable device, the preset relative position can be a relatively small value between 0 and 2 meters, which is consistent with the possible distance between the two devices in this case.

[0237] For example, if the first device is a movable device that is communicatively connected to the second device, the movable device is removably connected to a movable platform, and the relative position and relative posture between the movable device and the movable platform are fixed, the preset relative position can be a relatively large value between 1 and 8 meters. This is consistent with the possible distance between the two devices in this case.

[0238] Any method for relative positioning of a first device and a second device provided in the embodiments of the present disclosure can be performed by any appropriate device with data processing capabilities, including but not limited to a terminal device and a server. Alternatively, any method for relative positioning of a first device and a second device provided in the embodiments of the present disclosure can be performed by a processor, such as by invoking corresponding instructions stored in a memory to execute any method for relative positioning of a first device and a second device mentioned in the embodiments of the present disclosure. This will not be further described below.

[0239] Exemplary devices

[0240] As shown in Figure 17, it is a schematic diagram of the structure of an apparatus for relative positioning of a first device and a second device provided by some exemplary embodiments of the present disclosure. The first device is provided with a first inertial measurement unit and a reference object, and the second device is provided with a second inertial measurement unit and an image sensor. The apparatus includes: a first acquisition module 171 for acquiring a state quantity, first data collected by the first inertial measurement unit, second data collected by the second inertial measurement unit, and an image with the reference object collected by the image sensor; wherein the state quantity includes: the relative position and relative posture between the second device and the first device, the target parameter associated with the relative velocity between the second device and the first device, the angular velocity bias and acceleration bias of the first inertial measurement unit, and the angular velocity bias and acceleration bias of the second inertial measurement unit; a first determination module 173 for determining a priori state quantity based on the state quantity, the first data, and the second data; a second determination module 175 for determining image features of the image with the reference object; and a first correction module 177 for correcting the priori state quantity based on the image features to obtain a corrected state quantity.

[0241] In some optional embodiments of the present disclosure, the first correction module 177 includes: a first determination submodule, used to determine the observed relative position and observed relative posture between the second device and the first device based on image features; a first correction submodule, used to correct the prior state quantity by fusing the observed relative position, the observed relative posture and the prior state quantity to obtain a corrected state quantity.

[0242] In some optional embodiments of the present disclosure, as shown in Figure 18, the first correction module 177 includes: a second determination submodule 1771, which is used to determine the observed relative posture between the second device and the first device based on image features; a third determination submodule 1772, which is used to determine the prior relative posture between the second device and the first device based on the prior state quantity; a fourth determination submodule 1773, which is used to determine the first residual between the observed relative posture and the prior relative posture; and a second correction submodule 1774, which is used to use the first residual to correct the prior state quantity to obtain a corrected state quantity.

[0243] In some optional embodiments of the present disclosure, the first correction module 177 is configured to correct the priori state quantity by fusing the image feature and the priori state quantity to obtain a corrected state quantity.

[0244] In some optional embodiments of the present disclosure, as shown in FIG19 , the first correction module 177 includes: a fifth determination submodule 1775 for determining a first position of the reference object in the image captured by the image sensor based on image features; a first acquisition submodule 1776 for acquiring the optical center position of the image sensor in the coordinate system corresponding to the second device; a sixth determination submodule 1777 for determining a second position of the reference object in the image captured by the image sensor based on the prior relative position and prior relative posture in the prior state quantity, and the optical center position; a seventh determination submodule 1778 for determining a second residual between the first position and the second position; and a third correction submodule 1779 for correcting the prior state quantity using the second residual to obtain a corrected state quantity.

[0245] In some optional embodiments of the present disclosure, the second determination module 175 includes: an eighth determination submodule, used to determine a local area for searching a reference object from an image with a reference object based on a prior relative position and a prior relative posture in a priori state quantity; and a ninth determination submodule, used to determine image features of the local area.

[0246] In some optional embodiments of the present disclosure, the number of reference objects is two, and before obtaining the state quantity, the first data collected by the first inertial measurement unit, the second data collected by the second inertial measurement unit, and the image collected by the image sensor, as shown in FIG20 , the apparatus provided by an embodiment of the present disclosure further includes: a third determination module 2010 for determining the first observed gravity direction through the first inertial measurement unit; a fourth determination module 2020 for determining the second observed gravity direction through the second inertial measurement unit; a second acquisition module 2030 for obtaining two third positions corresponding to the two reference objects in the coordinate system corresponding to the first device. ; The third acquisition module 2040 is used to acquire the initialization image with two reference images captured by the image sensor; the fifth determination module 2050 is used to determine the two fourth positions corresponding to the two reference objects in the coordinate system corresponding to the second device based on the initialization image; the sixth determination module 2060 is used to determine the initialization relative position and the initialization relative posture between the first device and the second device based on the first observed gravity direction, the second observed gravity direction, the two third positions and the two fourth positions; wherein the initialization relative position is used as the initial value of the relative position in the state quantity, and the initialization relative posture is used as the initial value of the relative posture in the state quantity.

[0247] In some optional embodiments of the present disclosure, the sixth determining module 2060 includes: a tenth determining submodule for determining, based on the two third positions, a first vector pointing from one of the two reference objects to the other in a coordinate system corresponding to the first device; an eleventh determining submodule for determining, based on the two fourth positions, a second vector pointing from one of the two reference objects to the other in a coordinate system corresponding to the second device;

[0248] The twelfth determination submodule is used to determine the initialization relative posture based on the first observed gravity direction, the second observed gravity direction, the first vector and the second vector; the thirteenth determination submodule is used to determine the initialization relative position based on the two third positions, the two fourth positions and the initialization relative posture.

[0249] In some optional embodiments of the present disclosure, the target parameters include at least one of the following two: the relative speed between the second device and the first device, and the projection result of the relative speed between the second device and the first device in the inertial coordinate system projected onto the reference coordinate system, where the reference coordinate system is the coordinate system corresponding to one of the first inertial measurement unit and the second inertial measurement unit; the first determination module 173 includes: a second acquisition submodule, for acquiring a state estimation equation that meets a predetermined lever arm constraint condition; wherein the predetermined lever arm constraint condition is defined by at least one of the following first equation and second equation, the first equation defines the relative speed between the second device and the first device, and the second equation defines the relative position between the second device and the first device; a fourteenth determination submodule, for determining the prior state quantity based on the state quantity, the first data, the second data and the state estimation equation.

[0250] In some optional embodiments of the present disclosure, as shown in Figure 21, the device provided by the embodiment of the present disclosure also includes: a fourth acquisition module 2110, used to acquire an image captured by the image sensor; a seventh determination module 2120, used to determine the target relative position in response to the image captured by the image sensor satisfying a preset loss condition associated with the reference object; wherein the target relative position refers to: the corrected relative position in the most recent corrected state quantity obtained before the image captured by the image sensor satisfies the preset loss condition; a second correction module 2130, used to use the target relative position to correct the prior state quantity to obtain a corrected state quantity.

[0251] In some optional embodiments of the present disclosure, the second device includes: a head-mounted display device; as shown in Figure 22-1, the apparatus provided by an embodiment of the present disclosure also includes: a first rendering module 2210, which is used to correct the prior state quantity based on image features in the first correction module 177, and after obtaining the corrected state quantity, render the image to be displayed of the head-mounted display device based on the corrected relative position and corrected relative posture in the currently obtained corrected state quantity, so that the head-mounted display device can display the image to be displayed.

[0252] In some optional embodiments of the present disclosure, the second device includes: a head-mounted display device; as shown in Figure 22-2, the apparatus provided by an embodiment of the present disclosure also includes: a second rendering module 2220, which is used to correct the prior state quantity using the target relative position in the second correction module 2130, and after obtaining the corrected state quantity, render the image to be displayed of the head-mounted display device based on the target relative position and the corrected relative posture in the currently obtained corrected state quantity, so that the head-mounted display device can display the image to be displayed.

[0253] It can be understood that displaying the image to be displayed by the head-mounted display device may not be a step of the method in this embodiment, but other additional steps.

[0254] In some optional embodiments of the present disclosure, as shown in FIG23 , the first correction module 177 includes:

[0255] The fourth correction submodule 1780 is used to correct the prior state quantity based on the image features and using the iterative extended Kalman filter algorithm in response to the image with the reference object meeting the preset loss and recovery conditions associated with the reference object; the device provided by the embodiment of the present disclosure also includes: a switching module 2310, which is used to switch the algorithm used for correcting the prior state quantity from the iterative extended Kalman filter algorithm to a predetermined algorithm in response to the operating information of the iterative extended Kalman filter algorithm meeting the preset operating conditions.

[0256] In the device of the present disclosure, the various optional embodiments, optional implementation methods and optional examples disclosed above can be flexibly selected and combined as needed to achieve corresponding functions and effects, and the present disclosure does not list them one by one.

[0257] Exemplary electronic devices

[0258] FIG24 illustrates a block diagram of an electronic device according to an embodiment of the present disclosure. The electronic device 2400 includes one or more processors 2410 and a memory 2420 .

[0259] The processor 2410 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 2400 to perform desired functions.

[0260] The memory 2420 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 2410 may execute the one or more computer program instructions to implement the methods of the various embodiments of the present disclosure described above and / or other desired functions.

[0261] In one example, the electronic device 2400 may further include an input device 2430 and an output device 2440 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0262] The input device 2430 may also include, for example, a keyboard, a mouse, and the like.

[0263] The output device 2440 can output various information to the outside, and may include, for example, a display, a speaker, a printer, a communication network and its connected remote output device, etc.

[0264] Of course, for simplicity, FIG24 only shows some of the components related to the present disclosure in the electronic device 2400, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 2400 may further include any other appropriate components depending on the specific application.

[0265] Exemplary computer program products and computer-readable storage media

[0266] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions that, when executed by a processor, enable the processor to perform the steps of the method according to various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.

[0267] The computer program product may be written in any combination of one or more programming languages ​​to implement the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0268] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, causes the processor to execute the steps of the method according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0269] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0270] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and non-limiting, and should not be construed as necessarily possessed by each embodiment of the present disclosure. The specific details disclosed above are provided for illustrative purposes and to facilitate understanding, and are not intended to be limiting. The above details do not limit the present disclosure to necessarily being implemented using the above specific details.

[0271] Those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is intended to include these modifications and variations.

Claims

1. A method for relative positioning of a first device and a second device, where the first device is provided with a first inertial measurement unit and a reference object, and the second device is provided with a second inertial measurement unit and an image sensor. The method includes: Obtaining state quantities, first data collected by the first inertial measurement unit, second data collected by the second inertial measurement unit, and an image with the reference object collected by the image sensor; where the state quantities include: the relative position and relative attitude between the second device and the first device, target parameters associated with the relative velocity between the second device and the first device, the angular velocity bias and acceleration bias of the first inertial measurement unit, and the angular velocity bias and acceleration bias of the second inertial measurement unit; Determining a prior state quantity based on the state quantities, the first data, and the second data; Determining image features of the image with the reference object; Correcting the prior state quantity based on the image features to obtain a corrected state quantity.

2. The method according to claim 1, wherein, The correcting the prior state quantity based on the image features to obtain a corrected state quantity includes: Determining an observed relative position and an observed relative attitude between the second device and the first device based on the image features; Correcting the prior state quantity by fusing the observed relative position, the observed relative attitude, and the prior state quantity to obtain a corrected state quantity.

3. The method according to claim 1, wherein, The correcting the prior state quantity based on the image features to obtain a corrected state quantity includes: Determining an observed relative pose between the second device and the first device based on the image features; Determining a prior relative pose between the second device and the first device based on the prior state quantity; Determining a first residual between the observed relative pose and the prior relative pose; Correcting the prior state quantity using the first residual to obtain a corrected state quantity.

4. The method according to claim 1, wherein The correcting the prior state quantity based on the image features to obtain a corrected state quantity includes: Correcting the prior state quantity by fusing the image features and the prior state quantity to obtain a corrected state quantity.

5. The method according to claim 1, wherein, The correcting the prior state quantity based on the image features to obtain a corrected state quantity includes: Determining a first position of the reference object in the image collected by the image sensor based on the image features; Obtaining the optical center position of the image sensor in the coordinate system corresponding to the second device; Determining a second position of the reference object in the image collected by the image sensor based on the prior relative position and prior relative attitude in the prior state quantity, and the optical center position; Determining a second residual between the first position and the second position; Correcting the prior state quantity using the second residual to obtain a corrected state quantity.

6. The method according to any one of claims 1 to 5, wherein, The determining the image features of the image with the reference object includes: Determining a local area for searching the reference object from the image with the reference object based on the prior relative position and prior relative attitude in the prior state quantity; Determine the image features of the local area.

7. The method according to any one of claims 1 to 6, wherein The number of the reference objects is two. Before acquiring the state quantity, the first data collected by the first inertial measurement unit, the second data collected by the second inertial measurement unit, and the image collected by the image sensor, the method further includes: Determine a first observed gravity direction through the first inertial measurement unit; Determine a second observed gravity direction through the second inertial measurement unit; Acquire two third positions corresponding to the two reference objects in the coordinate system corresponding to the first device; Acquire an initialization image with the two reference images collected by the image sensor; Based on the initialization image, determine two fourth positions corresponding to the two reference objects in the coordinate system corresponding to the second device; Based on the first observed gravity direction, the second observed gravity direction, the two third positions, and the two fourth positions, determine an initialization relative position and an initialization relative attitude between the first device and the second device; Wherein, the initialization relative position serves as an initial value of the relative position in the state quantity, and the initialization relative attitude serves as an initial value of the relative attitude in the state quantity.

8. The method according to claim 7, wherein The determining the initialization relative position and the initialization relative attitude between the first device and the second device based on the first observed gravity direction, the second observed gravity direction, the two third positions, and the two fourth positions includes: Based on the two third positions, determine a first vector pointing from one of the two reference objects to the other in the coordinate system corresponding to the first device; Based on the two fourth positions, determine a second vector pointing from one of the two reference objects to the other in the coordinate system corresponding to the second device; Based on the first observed gravity direction, the second observed gravity direction, the first vector, and the second vector, determine the initialization relative attitude; Based on the two third positions, the two fourth positions, and the initialization relative attitude, determine the initialization relative position.

9. The method according to any one of claims 1 to 8, wherein, The target parameter includes at least one of the following two: the relative velocity between the second device and the first device, and a projection result obtained by projecting the relative velocity between the second device and the first device in an inertial coordinate system onto a reference coordinate system, where the reference coordinate system is the coordinate system corresponding to one of the first inertial measurement unit and the second inertial measurement unit; The determining the prior state quantity based on the state quantity, the first data, and the second data includes: Acquire a state estimation equation that meets a predetermined lever arm constraint condition; wherein, the predetermined lever arm constraint condition is defined by at least one of the following first equation and second equation, the first equation defines the relative velocity between the second device and the first device, and the second equation defines the relative position between the second device and the first device; Based on the state quantity, the first data, the second data, and the state estimation equation, determine the prior state quantity.

10. The method according to any one of claims 1 to 9, wherein, The method further includes: Acquire the image collected by the image sensor; Determine a target relative position in response to the image acquired by the image sensor satisfying a preset loss condition associated with the reference object; wherein the target relative position refers to one of the following two: a preset relative position, and a corrected relative position in the most recent corrected state quantity obtained before the image acquired by the image sensor satisfies the preset loss condition. Use the target relative position to correct the prior state quantity to obtain a corrected state quantity.

11. The method according to claim 10, wherein, The second device includes: a head-mounted display device. The method further includes one of the following two: After correcting the prior state quantity based on the image feature to obtain a corrected state quantity, render a to-be-displayed image of the head-mounted display device based on the corrected relative position and corrected relative attitude in the currently obtained corrected state quantity, for the head-mounted display device to display the to-be-displayed image. After using the target relative position to correct the prior state quantity to obtain a corrected state quantity, render the to-be-displayed image of the head-mounted display device based on the target relative position and the corrected relative attitude in the currently obtained corrected state quantity.

12. The method according to any one of claims 1 to 11, wherein, The correcting the prior state quantity based on the image feature to obtain a corrected state quantity includes: In response to an image with the reference object satisfying a preset loss recovery condition associated with the reference object, correct the prior state quantity based on the image feature by using an iterative extended Kalman filter algorithm. The method further includes: In response to the operation information of the iterative extended Kalman filter algorithm satisfying a preset operation condition, switch the algorithm for correcting the prior state quantity from the iterative extended Kalman filter algorithm to a predetermined algorithm.

13. The method according to claim 10, wherein, The first device is a movable device communicatively connected to the second device, the movable device is used to control the second device, the second device is a head-mounted display device, and the preset relative position is one of the following two: The preset relative position is a fixed value from 0 to 2 meters. The movable device is removably connected to a movable platform where the head-mounted display device is located, and the preset relative position is a fixed value from 1 to 8 meters.

14. An electronic device, comprising: A memory for storing a computer program product; A processor for executing the computer program product stored in the memory, and when the computer program product is executed, implementing the method for relative positioning of the first device and the second device according to any one of claims 1 to 13 above.

15. A computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, implementing the method for relative positioning of the first device and the second device according to any one of claims 1 to 13 above.

Images