Method and device for screening out position of exterior point, electronic equipment and storage medium

Abstract

The invention provides a method and device for screening out an exterior point position, electronic equipment and a storage medium. According to the specific implementation scheme, a first observation gravity direction is determined through a first inertial measurement unit arranged on first equipment; wherein the first equipment is provided with a reference object; determining a second observation gravity direction through a second inertial measurement unit arranged on second equipment; wherein the second equipment is provided with an image sensor; determining a candidate position set of the reference object from the image acquired by the image sensor; acquiring an optical center position of the image sensor under a coordinate system corresponding to the second equipment; and based on the first observation gravity direction, the second observation gravity direction and the optical center position, performing exterior point position screening processing on the candidate position set.

Description

Technical Field

[0001] The present disclosure relates to the technical field of relative positioning, and particularly to a method, an apparatus, an electronic device, and a storage medium for screening outlier positions. Background Art

[0002] In some application scenarios, relative positioning of two devices is required without paying attention to the absolute positioning of each of the two devices. Summary of the Invention

[0003] According to one aspect of the embodiments of the present disclosure, a method for screening outlier positions is provided, including: determining a first observed gravity direction through a first inertial measurement unit disposed on a first device, where the first device is provided with a reference object; determining a second observed gravity direction through a second inertial measurement unit disposed on a second device, where the second device is provided with an image sensor; determining a candidate position set of the reference object from an image collected by the image sensor; obtaining the optical center position of the image sensor in the coordinate system corresponding to the second device; and performing outlier position screening processing on the candidate position set based on the first observed gravity direction, the second observed gravity direction, and the optical center position.

[0004] According to another aspect of the embodiments of the present disclosure, an apparatus for screening outlier positions is provided, including: a first determination module for determining a first observed gravity direction through a first inertial measurement unit disposed on a first device, where the first device is provided with a reference object; a second determination module for determining a second observed gravity direction through a second inertial measurement unit disposed on a second device, where the second device is provided with an image sensor; a third determination module for determining a candidate position set of the reference object from an image collected by the image sensor; an obtaining module for obtaining the optical center position of the image sensor in the coordinate system corresponding to the second device; and a processing module for performing outlier position screening processing on the candidate position set based on the first observed gravity direction, the second observed gravity direction, and the optical center position.

[0005] According to still another aspect of the present disclosure, a computer-readable storage medium is provided, and the storage medium stores a computer program for executing the above method for screening outlier positions.

[0006] According to yet another aspect of the present disclosure, an electronic device is provided, including: a processor; a memory for storing processor-executable instructions; and the processor for reading the executable instructions from the memory and executing the instructions to implement the above method for screening outlier positions. Brief Description of the Drawings

[0007] Figure 1 It is a schematic diagram of an application scenario of some exemplary embodiments of the present disclosure.

[0008] Figure 2 It is a schematic flowchart of a method for screening outlier positions provided by some exemplary embodiments of the present disclosure.

[0009] Figure 3 It is a schematic diagram of the first coordinate system and the second coordinate system in some exemplary embodiments of the present disclosure.

[0010] Figure 4 It is a schematic flowchart of a method for screening outlier positions provided by some other exemplary embodiments of the present disclosure.

[0011] Figure 5 It is a schematic diagram of applying the first preset constraint condition in some exemplary embodiments of the present disclosure.

[0012] Figure 6 It is a schematic flowchart of a method for determining whether there is a second vector in a target vector set provided by some exemplary embodiments of the present disclosure.

[0013] Figure 7-1 It is a schematic flowchart of a method for screening outlier positions from a candidate position set based on a first judgment result provided by some exemplary embodiments of the present disclosure.

[0014] Figure 7-2 It is a schematic flowchart of a method for screening outlier positions from a candidate position set based on a first judgment result provided by some other exemplary embodiments of the present disclosure.

[0015] Figure 8 It is a schematic flowchart of a method for determining the relative attitude between a first device and a second device based on a first observed gravity direction, a second observed gravity direction, a first vector, and a second vector provided by some exemplary embodiments of the present disclosure.

[0016] Figure 9 It is a schematic flowchart of a method for determining the relative attitude between a first device and a second device based on a first observed gravity direction, a second observed gravity direction, a first vector, and a second vector provided by some other exemplary embodiments of the present disclosure.

[0017] Figure 10-1 It is a schematic diagram of determining a first unit vector in some exemplary embodiments of the present disclosure.

[0018] Figure 10-2 It is a schematic diagram of determining a second unit vector in some exemplary embodiments of the present disclosure.

[0019] Figure 11 It is a schematic flowchart of a method for determining whether a first candidate position pair satisfies a second preset constraint condition based on a relative attitude and an optical center position to obtain a second judgment result provided by some exemplary embodiments of the present disclosure.

[0020] Figure 12 It is a schematic diagram of two surfaces of the first device arranged opposite to each other in some exemplary embodiments of the present disclosure.

[0021] Figure 13 It is a schematic flowchart of a method for determining whether a first candidate position pair satisfies a second preset constraint condition based on a preset position relationship, a fourth vector, a fifth vector, and a sixth vector to obtain a second determination result provided in some exemplary embodiments of the present disclosure.

[0022] Figure 14 It is a schematic diagram of the principle of applying the second preset constraint condition in some exemplary embodiments of the present disclosure.

[0023] Figure 15 It is a schematic flowchart of a method for screening outlier positions provided in some other exemplary embodiments of the present disclosure.

[0024] Figure 16 It is a schematic structural diagram of a device for screening outlier positions provided in some exemplary embodiments of the present disclosure.

[0025] Figure 17 It is a schematic structural diagram of a processing module in some exemplary embodiments of the present disclosure.

[0026] Figure 18 It is a schematic diagram of a module related to determining whether a second vector exists in a target vector set in some exemplary embodiments of the present disclosure.

[0027] Figure 19 It is a schematic structural diagram of a first processing sub-module in some exemplary embodiments of the present disclosure.

[0028] Figure 20 It is a schematic structural diagram of a first processing sub-module in some other exemplary embodiments of the present disclosure.

[0029] Figure 21 It is a schematic structural diagram of a processing module in some other exemplary embodiments of the present disclosure.

[0030] Figure 22 It is a schematic structural diagram of a second determination sub-module in some exemplary embodiments of the present disclosure.

[0031] Figure 23 It is a structural diagram of an electronic device provided in some exemplary embodiments of the present disclosure. Detailed implementation manners

[0032] To explain the present disclosure, exemplary embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all embodiments. It should be understood that the present disclosure is not limited by the exemplary embodiments.

[0033] It should be noted that: Unless otherwise specifically stated, the relative arrangements of components and steps, numerical expressions, and numerical values set forth in these embodiments do not limit the scope of the present disclosure.

[0034] Exemplary method

[0035] In some application scenarios, relative positioning of two devices is required.

[0036] For example, one of the two devices can be a head-mounted display device, and the other can be a movable platform on which the head-mounted display device is located. The head-mounted display device can also be referred to as a Head-Mounted Display (HMD) or a head-mounted display. The head-mounted display device can be presented in the form of glasses, helmets, etc. The head-mounted display device can include, but is not limited to, Augmented Reality (AR) glasses, Virtual Reality (VR) glasses, etc. The movable platform can include, but is not limited to, vehicles, ships, airplanes, trains, etc. By relatively positioning the head-mounted display device and the movable platform, the relative pose between the head-mounted display device and the movable platform can be obtained. Referring to the relative pose between the head-mounted display device and the movable platform, the virtual image displayed by the head-mounted display device can be rendered, adjusted, warped, controlled, etc. to achieve a specific display effect, such as achieving a 6dof (degree of freedom) display effect.

[0037] For another example, one of the two devices can be a drone, and the other can be a movable platform to which the drone is to land. By relatively positioning the drone and the movable platform, the relative pose between the drone and the movable platform can be obtained. Referring to the relative pose between the drone and the movable platform, the motion parameters of the drone and the movable platform can be controlled respectively to enable the drone to land on the movable platform smoothly.

[0038] For yet another example, one of the two devices can be a first aircraft flying in the air and in need of refueling, and the other can be a second aircraft flying in the air and used for fuel supply. By relatively positioning the first aircraft and the second aircraft, the relative pose between the first aircraft and the second aircraft can be obtained. Referring to the relative pose between the first aircraft and the second aircraft, the flight parameters of the first aircraft and the second aircraft can be controlled respectively to enable the second aircraft to refuel the first aircraft smoothly in the air.

[0039] Of course, the application scenarios that require relative positioning of two devices are not limited to this, and will not be enumerated one by one here. For ease of description, in the following text, one of the two devices that requires relative positioning may be referred to as the first device, and the other as the second device.

[0040] In some alternative embodiments of the present disclosure, the first device may be a movable platform where the head-mounted display device is located, and the second device may be the head-mounted display device.

[0041] In some other alternative embodiments of the present disclosure, the first device may be a portable device removably fixed to the movable platform where the head-mounted display device is located, and the second device may be the head-mounted display device. The removable fixing method may include but is not limited to screwing, clamping, plugging, etc. Since the portable device and the movable platform are relatively fixed, as long as the relative positioning between the head-mounted display device and the portable device is achieved, the relative positioning between the head-mounted display device and the movable platform can be achieved on this basis.

[0042] The first device and the second device may be respectively provided with an Inertial Measurement Unit (IMU). For ease of distinction, the inertial measurement unit provided in the first device may be referred to as the first inertial measurement unit, and the inertial measurement unit provided in the second device may be referred to as the second inertial measurement unit. It can be understood that the inertial measurement unit is a device that uses an accelerometer and a gyroscope to measure the angular velocity and acceleration of an object. In this way, the first inertial measurement unit can be used to measure the angular velocity and acceleration of the first device in the world coordinate system, and the second inertial measurement unit can be used to measure the angular velocity and acceleration of the second device in the world coordinate system. The world coordinate system can be understood as a reference reference coordinate system, and the present disclosure does not limit the specific setting of the world coordinate system.

[0043] In addition to being provided with the first inertial measurement unit, the first device may also be provided with a reference object. The reference object may be a natural feature existing in the first device itself, such as a corner point, a bump, etc. on the first device. Or, the reference object may be a marker additionally added to assist in realizing the relative positioning between the first device and the second device, such as an object with a special color or pattern, or a Light Emitting Diode (LED) lamp, etc. The number of reference objects may be two, three, or more than three, and will not be enumerated one by one here. In an optional example, as Figure 1 shown, the second device may be a head-mounted display device worn by the user, the number of reference objects may be two, and the two reference objects may be the first marker 110 and the second marker 120, and the first marker 110 and the second marker 120 may be arranged vertically.

[0044] In addition to being provided with a second inertial measurement unit, the second device may also be provided with an image sensor. The image sensor may be a sensor for collecting images. The light incident surface of the image sensor may face the first device so as to collect images. It can be understood that the image sensor may collect images including the reference object, or may not collect images including part of the reference object or not including the reference object.

[0045] In the embodiments of the present disclosure, the gravity direction in the real physical world can be observed respectively through the first inertial measurement unit and the second inertial measurement unit to obtain the observation results corresponding to the first inertial measurement unit and the second inertial measurement unit respectively. In addition, images can be collected through the image sensor, and a candidate position set of the reference object can be determined from the images collected by the image sensor. The candidate position set may include multiple candidate positions where the reference object may exist in the image. According to the observation results corresponding to the first inertial measurement unit and the second inertial measurement unit respectively, outlier position screening processing can be performed on the candidate position set to exclude candidate positions belonging to outlier positions from the candidate position set, and on this basis, determine candidate positions belonging to inlier positions. An outlier position can be understood as: a position in the image that is not the true position of the reference object. An inlier position can be understood as: the true position of the reference object in the image. The inlier position can be used for relative positioning between the first device and the second device.

[0046] In some alternative embodiments of the present disclosure, the reference object may have a relatively regular shape, and the position of the reference object in the image can be characterized by the coordinates of the geometric center of the reference object.

[0047] As Figure 2 shown, it is a schematic flowchart of a method for screening outlier positions provided by some exemplary embodiments of the present disclosure. Figure 2 The method shown may include step 210, step 220, step 230, step 240, and step 250.

[0048] Step 210, determine a first observed gravity direction through a first inertial measurement unit provided in the first device.

[0049] In step 210, the gravity direction in the real physical world can be observed through the first inertial measurement unit to obtain the first observed gravity direction.

[0050] In some alternative embodiments of the present disclosure, the gravity direction in the real physical world can be represented as Figure 3 the direction indicated by g in

[0051] In some alternative embodiments of the present disclosure, when the first device is in a stationary state, the acceleration of the first device can be measured by the first inertial measurement unit. The direction of the measured acceleration can be regarded as an observation result of the direction of gravity in the real physical world. Then, the direction of the measured acceleration can be used as the first observed gravity direction. Optionally, the first observed gravity direction can be in vector form.

[0052] It should be noted that when the first device is in a moving state, the direction of gravity in the real physical world can also be observed. However, to ensure the observation accuracy, the acceleration generated by the movement of the first device cannot be too large.

[0053] Step 220: Determine the second observed gravity direction through the second inertial measurement unit provided in the second device.

[0054] In step 220, the direction of gravity in the real physical world can be observed through the second inertial measurement unit to obtain the second observed gravity direction. The specific method for obtaining the second observed gravity direction can refer to the relevant introduction of step 210 above and will not be elaborated here. Similar to the first observed gravity direction, the second observed gravity direction can also be in vector form.

[0055] Step 230: Determine a set of candidate positions of the reference object from the images collected by the image sensor.

[0056] In some alternative embodiments of the present disclosure, the images collected by the image sensor can be processed such as feature extraction and recognition to determine the pixel points that may be the reference object from the images. The set of the positions of these determined pixel points can be used as the candidate position set.

[0057] Step 240: Obtain the optical center position of the image sensor in the coordinate system corresponding to the second device.

[0058] In some alternative embodiments of the present disclosure, both the first device and the second device can have corresponding coordinate systems. The coordinate system corresponding to the first device can be called the first coordinate system. The coordinate system corresponding to the second device can be called the second coordinate system. The first coordinate system can refer to a three-dimensional coordinate system constructed with the centroid or other position points of the first device as the origin. The second coordinate system can refer to a three-dimensional coordinate system constructed with the centroid or other position points of the second device as the origin. In an alternative example, the first coordinate system can be Figure 3 with the origin O m , and the three coordinate axes are X m , Y m , Z m of the three-dimensional coordinate system, and the second coordinate system can be Figure 3 with the origin O b, the three coordinate axes are respectively X b , Y b , Z b in a three-dimensional coordinate system.

[0059] It should be noted that since the image sensor is disposed on the second device, the image sensor can be considered stationary in the second coordinate system, and the optical center position of the image sensor can be fixed and known.

[0060] Step 250: Based on the first observed gravity direction, the second observed gravity direction, and the optical center position, perform an outlier position screening process on the candidate position set.

[0061] In some alternative embodiments of the present disclosure, constraint conditions for screening outlier positions can be preset in advance. The number of constraint conditions can be one, two, or more than two, and the present disclosure does not limit the specific number of constraint conditions. In step 250, in combination with the first observed gravity direction, the second observed gravity direction, and the optical center position, it can be determined which candidate positions in the candidate position set conform to the constraint conditions and which candidate positions do not conform to the constraint conditions, and accordingly, the candidate positions belonging to the outlier positions can be excluded from the candidate position set.

[0062] In the embodiments of the present disclosure, the first inertial measurement unit and the second inertial measurement unit can be used to respectively observe the gravity direction in the real physical world to obtain the first observed gravity direction and the second observed gravity direction. Additionally, the positions where reference objects may exist in the image can be determined through the images collected by the image sensor to obtain the candidate position set. The first observed gravity direction, the second observed gravity direction, and the optical center position of the image sensor can be used for the outlier position screening process of the candidate position set. In this way, the outlier positions can be effectively excluded from the candidate position set, avoiding the adverse effects of the outlier positions on the relative positioning between the first device and the second device, and thus being able to better ensure the relative positioning accuracy between the first device and the second device.

[0063] In some alternative embodiments of the present disclosure, the first device can be provided with two reference objects. For the convenience of description, one of the two reference objects can be referred to as the first reference object hereinafter, and the other can be referred to as the second reference object.

[0064] As Figure 4 shown, it is a schematic flowchart of a method for screening outlier positions provided by some other exemplary embodiments of the present disclosure. Figure 4 The method shown can include step 410, step 420, step 430, and step 440. Optionally, the combination of step 410 to step 440 can be used as an alternative embodiment of step 250 of the present disclosure.

[0065] Step 410: Determine a first vector in the coordinate system corresponding to the first device, which points from one of the two reference objects to the other.

[0066] As introduced above, the coordinate system corresponding to the first device can also be referred to as the first coordinate system. Since both reference objects are set on the first device, they can be considered stationary in the first coordinate system, and their coordinates in the first coordinate system can be fixed and known. Based on the coordinates of the two reference objects in the first coordinate system, the first vector can be determined. The first vector can be either the vector pointing from the first reference object to the second reference object in the first coordinate system or the vector pointing from the second reference object to the first reference object in the first coordinate system. In an optional example, in the first coordinate system, the coordinates of the first reference object are (x1, y1, z1), and the coordinates of the second reference object are (x2, y2, z2). Then, the first vector can be (x2 - x1, y2 - y1, z2 - z1), or (x1 - x2, y1 - y2, z1 - z2).

[0067] Optionally, before performing Figure 2 the method shown, before step 250, it is also possible to obtain the coordinates of the first reference object and the coordinates of the second reference object in the coordinate system corresponding to the first device. Then, based on the coordinates of the first reference object, the coordinates of the second reference object, the first observed gravity direction, the second observed gravity direction, and the optical center position, outlier position screening processing is performed on the candidate position set.

[0068] Step 420: Determine a first candidate position pair including a first candidate position and a second candidate position from the candidate position set.

[0069] In some optional embodiments of the present disclosure, two candidate positions can be randomly selected from the candidate position set, and one of the two selected candidate positions is used as the first candidate position, and the other is used as the second candidate position, thereby obtaining a first candidate position pair including the first candidate position and the second candidate position.

[0070] Step 430: Based on the first observed gravity direction, the second observed gravity direction, the optical center position, and the first vector, determine whether the first candidate position pair satisfies a first preset constraint condition, and obtain a first judgment result.

[0071] In some optional embodiments of the present disclosure, step 430 may include:

[0072] In response to the existence of a second vector in the target vector set, obtain a first judgment result indicating that the first candidate position pair satisfies the first preset constraint condition;

[0073] In response to the non - existence of a second vector in the target vector set, obtain a first judgment result used to characterize that the first candidate position pair does not satisfy the first preset constraint condition;

[0074] Among them, the target vector set includes: vectors with a starting point located in one of the first ray and the second ray and an ending point located in the other of the first ray and the second ray. The first ray refers to: a ray with a starting point at the optical center position and passing through the first candidate position. The second ray refers to: a ray with a starting point at the optical center position and passing through the second candidate position. The second vector refers to: a vector whose angle with the second observed gravity direction is equal to the target angle. The target angle refers to: the angle between the first vector and the first observed gravity direction.

[0075] It should be noted that if there is a second vector in the target vector set, the starting point of the second vector can be the same as the starting point of the first vector, and the ending point of the second vector can be the same as the ending point of the first vector. For example, if the starting point of the first vector corresponds to the first reference object and the ending point of the first vector corresponds to the second reference object, then: the starting point of the second vector corresponds to the first reference object and the ending point of the second vector corresponds to the second reference object.

[0076] In an optional example, as Figure 5 shown, the optical center position of the image sensor can be represented as O b , the first candidate position can be represented as A b , the second candidate position can be represented as B b . The first ray can be a ray with a starting point at O b and passing through A b . The second ray can be a ray with a starting point at O b and passing through B b . The target vector set can include all vectors with a starting point located in one of the first ray and the second ray and an ending point located in the other of the first ray and the second ray. The first observed gravity direction can be represented as G m , the second observed gravity direction can be represented as G b , and the first vector can be represented as (AB) m .

[0077] It should be noted that theoretically, if both the first candidate position and the second candidate position are interior point positions, that is, the first candidate position is the true position of one of the first reference object and the second reference object in the image, and the second candidate position is the true position of the other of the first reference object and the second reference object in the image, then there must be a vector in the target vector set whose angle with G b is equal to the angle between (AB) m and G m , and this vector is the second vector.

[0078] In view of this, it is possible to determine whether there is a second vector in the target vector set. If there is a second vector in the target vector set, it can be considered that both the first candidate position and the second candidate position may belong to the inlier positions. Then, it can be determined that the first candidate position pair satisfies the first preset constraint condition, and a judgment result for characterizing that the first candidate position pair satisfies the first preset constraint condition is obtained. If there is no second vector in the target vector set, it can be considered that at least one of the first candidate position and the second candidate position belongs to the outlier position. Then, it can be determined that the first candidate position pair does not satisfy the first preset constraint condition, and a judgment result for characterizing that the first candidate position pair does not satisfy the first preset constraint condition is obtained.

[0079] Step 440, based on the first judgment result, perform an outlier position screening process on the candidate position set.

[0080] Next, an example of how to determine whether there is a second vector in the target vector set will be introduced.

[0081] As Figure 6 shown, it is a flowchart of a method for determining whether there is a second vector in the target vector set provided by some exemplary embodiments of the present disclosure. Figure 6 The method shown may include Step 610, Step 620, and Step 630.

[0082] Step 610, determine whether the independent variable x in the target equation has a solution; if the judgment result is yes, execute Step 620; if the judgment result is no, execute Step 630.

[0083] Step 620, determine that there is a second vector in the target vector set.

[0084] Step 630, determine that there is no second vector in the target vector set.

[0085] Among them, the target equation may include:

[0086] Ex 2 + Fx + H = 0, x ∈ (0, 1)

[0087] Among them,

[0088] E = [(b - a) · G b 2 - [(AB) m · G m 2 ||b - a|| 2

[0089] F = 2{(a · G b )[(b - a) · G b - [(AB) m · G​​m 2 [a·(b-a)]}

[0090] H=(a·G b ) 2 -[(AB) m ·G m 2 ||a|| 2

[0091] a represents a unit vector in the same direction as the third vector, where the third vector is the vector pointing from the first candidate position to the optical center position; b represents a unit vector in the same direction as the fourth vector, where the fourth vector is the vector pointing from the optical center position to the second candidate position; (AB) m represents the first vector, and G m represents the first observed gravity direction, and G b represents the second observed gravity direction.

[0092] Combined with Figure 5 , the second vector can be the vector with the starting point at A and the ending point at B. The unit vector in the same direction as vector A b O b can be represented as a. The unit vector in the same direction as vector O b B b can be represented as b. If A extends infinitely far on the first ray, the second vector can be in the same direction as a. If B extends infinitely far on the second ray, the second vector can be in the same direction as b. Thus, if the second vector is represented as c, c can satisfy the following first condition:

[0093] c = [a + x*(b - a)] / ||a + x*(b - a)||, x ∈ (0, 1)

[0094] In addition, c also satisfies the following second condition:

[0095] c·G b = (AB) m ·G m

[0096] Substitute the expression for the first condition into the above expression for the second condition, then we have:

[0097] [a + x*(b - a)] / ||a + x*(b - a)||·G b = (AB) m ·G m

[0098] [a + x*(b - a)]·G b = (AB) m ·G m ​​·||a + x*(b-a)||

[0099] {[a + x*(b-a)]·G b} 2 =[(AB) m ·G m 2 ·[a + x*(b-a)]·[a + x*(b-a)]

[0100] By further arranging the above formula, we can obtain:

[0101] {[(b-a)·G b 2 -[(AB) m ·G m 2 ||b-a|| 2}x 2 +2{(a·G b )[(b-a)·G b -[(AB) m ·G m 2 [a·(b-a)]}x+(a·G b ) 2 -[(AB) m ·G m 2 ||a|| 2 =0, x∈(0, 1)

[0102] The formula obtained after further arrangement is equivalent to the target equation in the above text.

[0103] It should be noted that the target equation is a quadratic equation with one variable whose independent variable is x, and x is within the range of (0, 1). Then, it can be determined whether there is a solution for the independent variable x in the target equation.

[0104] If there is a solution for the independent variable x in the target equation, it can be determined that there is a second vector in the target vector set, and a first judgment result for characterizing that the first candidate position pair satisfies the first preset constraint condition can be obtained. Optionally, if the second vector can be expressed as c, then c can be calculated using the following formula:

[0105] c = [a + x*(b-a)] / ||a + x*(b-a)||

[0106] It should be noted that if the number of solutions for the independent variable x in the target equation is only one, using the formula in the previous paragraph, one c can be determined. If the number of solutions for the independent variable x in the target equation is more than one (for example, two), using the formula in the previous paragraph, more than one c can be determined. ​​​​​

[0107] If the independent variable x in the target equation has no solution, it can be determined that there is no second vector in the target vector set, and a first judgment result for characterizing that the first candidate position pair does not meet the first preset constraint condition is obtained.

[0108] Figure 6 In the illustrated embodiment, the problem of determining whether there is a second vector in the target vector set can be transformed into the problem of determining whether the independent variable in the target equation has a solution. In this way, a complex problem can be simplified, and it can be determined efficiently and reliably whether there is a second vector in the target vector set, and thus it can be determined efficiently and reliably whether the first candidate position pair meets the first preset constraint condition.

[0109] For example, according to c·G b =(AB) m ·G m Based on this formula and the two preconditions that A is on the first ray and B is on the second ray, it is also possible to determine whether there is a second vector in the target vector set by other methods, so as to determine efficiently and reliably whether the first candidate position pair meets the first preset constraint condition.

[0110] Combined with the above introduction, it can be considered that the first preset constraint condition is an included angle constraint condition. Under the action of the included angle constraint condition, it can be effectively determined whether there is a second vector that meets the requirements in the target vector set, and based on this, the first judgment result can be obtained efficiently and reliably.

[0111] Such as Figure 7-1 shown, is a schematic flowchart of a method for performing outlier position screening processing on a candidate position set based on the first judgment result provided by some exemplary embodiments of the present disclosure. Figure 7-1 The method shown may include step 710, step 720, and step 730.

[0112] Step 710, in response to the first judgment result being used to characterize that the first candidate position pair does not meet the first preset constraint condition, determine a second candidate position pair including the first candidate position and a third candidate position from the candidate position set.

[0113] In some alternative embodiments of the present disclosure, if the first judgment result is used to characterize that the first candidate position pair does not meet the first preset constraint condition, a third candidate position different from the first candidate position and the second candidate position may be determined from the candidate position set, and the third candidate position and the first candidate position may be combined into a second candidate position pair.

[0114] Step 720, based on the first observed gravity direction, the second observed gravity direction, the optical center position, and the first vector, determine whether the second candidate position pair meets the first preset constraint condition to obtain a third judgment result.

[0115] It should be noted that the method for determining whether the second candidate position pair meets the first preset constraint condition can refer to the relevant introduction of the method for determining whether the first candidate position pair meets the first preset constraint condition, which will not be elaborated here.

[0116] Step 730: Based on the third judgment result, perform an outlier position screening process on the candidate position set.

[0117] As introduced above, the first candidate position and the second candidate position can be combined to obtain a first candidate position pair, and it can be determined whether the first candidate position pair meets the first preset constraint condition. In addition, the first candidate position and the third candidate position can be combined to obtain a second candidate position pair, and it can be determined whether the second candidate position pair meets the first preset constraint condition. Similarly, the first candidate position can also be combined with other candidate positions in the candidate position set, and it can be determined whether the other combination results meet the first preset constraint condition. If the first candidate position does not meet the first preset constraint condition no matter which candidate position it is combined with, it can be determined that the first candidate position belongs to an outlier position. Then, the first candidate position can be deleted from the candidate position set. If the combination result obtained by combining the first candidate position with some candidate positions meets the first preset constraint condition, other constraint conditions (such as the second preset constraint condition in the following text) can be combined to further determine whether the first candidate position belongs to an outlier position. In this way, outlier positions can be effectively screened from the candidate position set.

[0118] As Figure 7-2 shown, it is a flowchart of a method for performing an outlier position screening process on a candidate position set based on the first judgment result provided by some other exemplary embodiments of the present disclosure. Figure 7-2 The method shown can include Step 740, Step 750, and Step 760.

[0119] Step 740: In response to the first judgment result indicating that the first candidate position pair meets the first preset constraint condition, determine the relative pose between the first device and the second device based on the first observed gravity direction, the second observed gravity direction, the first vector, and the second vector.

[0120] In some alternative embodiments of the present disclosure, the relative pose between the first device and the second device may refer to the pose of the second device in the first coordinate system.

[0121] After determining the relative pose, Step 750 can be executed. It should be noted that if the number of solutions of the independent variable x in the target equation determined above is more than one, for each solution, the corresponding relative pose can be determined respectively, and for each determined relative pose, Step 750 is executed.

[0122] Step 750: Based on the relative pose and the optical center position, determine whether the first candidate position pair satisfies the second preset constraint condition, and obtain a second judgment result.

[0123] After obtaining the second judgment result, step 760 can be executed.

[0124] Step 760: Based on the second judgment result, perform an outlier position screening process on the candidate position set.

[0125] It should be noted that the specific implementation manner of step 760 can refer to the relevant introduction of step 730 in the foregoing, and will not be elaborated here.

[0126] In some alternative embodiments of the present disclosure, some methods for determining the relative pose between the first device and the second device are also provided for use in relevant methods that require relative pose.

[0127] As Figure 8 shown, the relative pose between the first device and the second device can be determined based on the first observed gravity direction, the second observed gravity direction, the first vector, and the second vector. Figure 8 The method shown in

[0128] Step 810: Determine the first cross product result of the first observed gravity direction and the first vector.

[0129] Assume that the first observed gravity direction is represented as G m , and the first vector is represented as (AB) m , then the first cross product result can be represented as G m ×(AB) m .

[0130] Step 820: Determine the second cross product result of the second observed gravity direction and the second vector.

[0131] Assume that the second observed gravity direction is represented as G b , and the second vector is represented as (AB) b , then the second cross product result can be represented as G b ×(AB) b .

[0132] Step 830: Combine the first observed gravity direction, the first vector, and the first cross product result to obtain a first combined matrix.

[0133] Step 840: Combine the second observed gravity direction, the second vector, and the second cross product result to obtain a second combined matrix.

[0134] As introduced above, the first observed gravity direction can be expressed as G m , the first vector can be expressed as (AB) m , the first cross product result can be expressed as G m ×(AB) m , then, the first combined matrix can be expressed as [G m , (AB) m , G m ×(AB) m .

[0135] As introduced above, the second observed gravity direction can be expressed as G b , the second vector can be expressed as (AB) b , the second cross product result can be expressed as G b ×(AB) b , then, the second combined matrix can be expressed as [G b , (AB) b , G b ×(AB) b .

[0136] Of course, when combining the first observed gravity direction, the first vector, and the first cross product result, and when combining the second observed gravity direction, the second vector, and the second cross product result, the combination order is not limited to the above examples. For example, the first combined matrix can also be expressed as [G m ×(AB) m , G m , (AB) m , and correspondingly, the second combined matrix can also be expressed as [G b ×(AB) b , G b , (AB) b .

[0137] Step 850, determine the relative attitude using the inverse matrix of the first combined matrix and the second combined matrix.

[0138] Assume the first combined matrix is expressed as [G m , (AB) m , G m ×(AB) m , the second combined matrix is expressed as [G b , (AB) b , G b ×(AB) b , and the relative attitude is expressed as R bm , then the relative attitude can be determined using the following formula:

[0139] R bm =[Gb , (AB) b , G b ×(AB) b [G m , (AB) m , G m ×(AB) m -1

[0140] For the sake of easy understanding, the principle of the formula used to determine the relative attitude R bm in the above paragraph will be introduced below.

[0141] G m and G b can be considered as the expressions of the same gravity direction in the first coordinate system and the second coordinate system respectively. (AB) m and (AB) b can both be considered as the vectors from the reference object A to the reference object B, and can be considered as the expressions in the first coordinate system and the second coordinate system respectively. G m ×(AB) m and G b ×(AB) b can also be considered as the expressions of the same meaning in the first coordinate system and the second coordinate system respectively. Then, we can have:

[0142] R bm* G m = G b

[0143] R bm* (AB) m = (AB) b

[0144] R bm* [G m ×(AB) m = G b ×(AB) b

[0145] By arranging the above three equations, the following equation can be obtained:

[0146] R bm* [G m , (AB) m , G m ×(AB) m = [G b , (AB) b , G b ×(AB) b

[0147] By multiplying both sides of the equation in the above paragraph by [G​​m , (AB) m , G m ×(AB) m -1 , the above formula for determining the relative attitude R bm can be obtained.

[0148] Figure 8 In the embodiment shown, by combining the vector cross product operation, the operation of combining vectors into a matrix, and the operation logic such as the inverse matrix of the matrix, the relative attitude can be determined efficiently and reliably.

[0149] As Figure 9 shown, based on the first observed gravity direction, the second observed gravity direction, the first vector, and the second vector, the relative attitude between the first device and the second device can be determined. Figure 9 The method shown in

[0150] may include step 910, step 920, step 930, step 940, step 950, and step 960.

[0151] Assume that the first observed gravity direction is represented as G m , and the second observed gravity direction is represented as G b . Then, using the vector angle calculation method, the angle θ of the rotation from G m to G b can be calculated, and the rotation axis is determined as G m ×G b , and the rotation matrix with the rotation angle of θ is determined. The determined rotation matrix can be used as the first rotation matrix. The first rotation matrix is used to rotate G m to align with G b , which can be understood as follows: If the magnitudes of G m and G b are the same, through the first rotation matrix, G m can be rotated to completely coincide with G b ; if the magnitudes of G m and G b are different, through the first rotation matrix, G m can be rotated to have the same starting point and the same direction as G b . The first rotation matrix can be represented as R(θ), then R(θ) can satisfy:

[0152] G b = R(θ)G m

[0153] ​Step 920, determine a first unit vector in the same direction as the first component; wherein, the first component refers to the component of the first vector perpendicular to the first observed gravity direction.

[0154] In an optional example, as Figure 10-1 shown, the first vector can be expressed as (AB) m , the starting point of the first vector can be expressed as A m , the ending point of the first vector can be expressed as B m , the first observed gravity direction can be expressed as G m . Then, a perpendicular line can be drawn from A m to G m , and the foot of the perpendicular can be expressed as D m , then the component of the first vector perpendicular to the first observed gravity direction can be expressed as (AD) m , that is, the first component can be expressed as (AD) m . The plane where A m , B m , D m are located can be called the A m B m D m plane. Using the following formula, the normal vector n m of the A m B m D m plane can be calculated:

[0155] n m =(AB) m ×G m / ||(AB) m ×G m ||

[0156] Assume that the first unit vector in the same direction as the first component is expressed as d m , and d m can be calculated using the following formula:

[0157] d m =G m ×n m / ||G m ×n m ||

[0158] Step 930, determine a second unit vector in the same direction as the second component; wherein, the second component refers to the component of the second vector perpendicular to the second observed gravity direction.

[0159] In an optional example, as Figure 10-2 shown, the second vector can be expressed as (AB) b , the starting point of the second vector can be expressed as Ab The end point of the second vector can be represented as B b The second observed gravity direction can be represented as G b Then, it can be from A b to G b draw a perpendicular line, and the foot of the perpendicular can be represented as D b Then the second component can be represented as (AD) b A b 、B b 、D b The plane where the three are located can be called the A b B b D b plane. Using the following formula, the normal vector n of the A b B b D b plane can be calculated b :

[0160] n b =(AB) b ×G b / ||(AB) b ×G b ||

[0161] Assume that the second unit vector in the same direction as the second component is represented as d b The following formula can be used to calculate d b :

[0162] d b =G b ×n b / ||G b ×n b ||

[0163] Step 940, use the first rotation matrix to rotate the first unit vector to obtain the third unit vector

[0164] Assume that the third unit vector is represented as d m ', and the following formula can be used to calculate d m ':

[0165] d m ' = R(θ)d m

[0166] Step 950, determine the second rotation matrix for rotating the third unit vector to align with the second unit vector

[0167] In some alternative embodiments of the present disclosure, the vector angle calculation method can be used to calculate the angle α from d m ' to d b and determine that the rotation axis is Gb , a rotation matrix with a rotation angle of α, and the determined rotation matrix can be used as the second rotation matrix. The second rotation matrix is used to rotate d m ' to align with d b Alignment can be understood as: through the second rotation matrix, d m ' can be rotated to align with d b completely coincide. The second rotation matrix can be expressed as R(α), then R(α) can satisfy the following formula:

[0168] d b = R(α)d m ' = R(α)R(θ)d m

[0169] Step 960, use the first rotation matrix and the second rotation matrix to determine the relative pose.

[0170] It should be noted that d b and d m can be considered as the expressions of the same vector in the first coordinate system and the second coordinate system respectively. Assuming the relative pose is expressed as R bm , then d b and d m need to satisfy:

[0171] d b = R bm d m

[0172] Based on this, the following formula can be obtained:

[0173] R bm = R(α)R(θ)

[0174] Obviously, the relative pose can be determined by the following set of formulas:

[0175] G b = R(θ)G m

[0176] n m = (AB) m × G m / ||(AB) m × G m |

[0177] d m = G m × n m / ||G m × n m ||

[0178] n b = (AB) b × Gb / ||(AB) b ×G b ||

[0179] d b =G b ×n b / ||G b ×n b ||

[0180] d m '=R(θ)d m

[0181] d b =R(α)d m '

[0182] R bm =R(α)R(θ)

[0183] It should be noted that for a device equipped with an inertial measurement unit, when the device rotates around the gravity direction in the real physical world, the acceleration data measured by the inertial measurement unit does not change, and the rotation angle of the device around the gravity direction in the real physical world cannot be observed. However, when the inclination angle of the device with respect to the gravity direction in the real physical world changes, this change can be observed through the acceleration data. Figure 9 In the illustrated embodiment, the relative inclination angles (equivalent to θ in the above text) of the first device and the second device with respect to the gravity direction in the real physical world can be determined first based on the observation results obtained by observing the gravity direction in the real physical world using the first inertial measurement unit and the second inertial measurement unit respectively. Based on the obtained relative inclination angles, in combination with the first vector and the second vector, the rotation angles (equivalent to α in the above text) of the first device and the second device rotating around the gravity direction can be calculated. The relative inclination angles and the rotation angles can be used together to determine the relative pose. In this way, the relative pose can effectively reflect the relative rotation between the first device and the second device, and the reliability of the relative pose can be better guaranteed.

[0184] As Figure 11 shown, it is a flowchart of a method for determining whether a first candidate position pair satisfies a second preset constraint condition based on a relative pose and an optical center position to obtain a second judgment result provided by some exemplary embodiments of the present disclosure. Figure 11 In the method shown, two reference objects can be arranged on the first surface of the first device, and the specified coordinate axis of the coordinate system corresponding to the first device and the first surface can satisfy a preset positional relationship.

[0185] In some optional embodiments of the present disclosure, the first device may include two surfaces disposed opposite to each other, one of which may be used to set two reference objects and is a surface that can usually be captured by an image sensor, and this surface may be used as the first surface, and the other surface may be a surface that cannot be captured by the image sensor, and this surface may be referred to as the second surface later. In an optional example, the first surface may be Figure 12 The second surface may be Figure 12 Surface 1220 in.

[0186] In some optional implementations of the present disclosure, the specified coordinate axis of the coordinate system corresponding to the first device may be the Z axis of the first coordinate system.

[0187] In some optional embodiments of the present disclosure, the specified coordinate axis and the first surface may satisfy the preset position relationship in the following two cases. Case 1: The specified coordinate axis is perpendicular to the first surface, and the specified coordinate axis is consistent with the direction of the vector from the first surface to the second surface; Case 2: The specified coordinate axis is perpendicular to the first surface, and the specified coordinate axis is consistent with the direction of the vector from the second surface to the first surface. For the above case 1: The direction of the specified coordinate axis can be referred to Figure 12 The direction indicated by the arrow J1 in the figure. For the above case 2: specify the direction of the coordinate axis, see Figure 12 The direction indicated by the arrow J2.

[0188] Below Figure 11 The method shown is described in detail.

[0189] Figure 11 The method shown may include step 1110 , step 1120 , step 1130 , step 1140 , and step 1150 .

[0190] Step 1110 , determining a third vector for representing the specified coordinate axis.

[0191] In some optional embodiments of the present disclosure, the third vector can be expressed as Figure 3 Z m . Z m It can be (0, 0, z). z can be a value greater than or equal to 0. For example, z can be 1, 2, 3, 5, etc., which are not listed here one by one.

[0192] Step 1120: Using the relative posture, convert the third vector into a coordinate system corresponding to the second device to obtain a fourth vector.

[0193] Assume that the fourth vector is represented by Z mb , the relative posture is represented by R bm , the fourth vector can be determined using the following formula:

[0194] Z mb = R bm Z m

[0195] Step 1130, determine a fifth vector pointing from the first candidate position to the optical center position.

[0196] Step 1140, determine a sixth vector pointing from the second candidate position to the optical center position.

[0197] It should be noted that specific examples for determining the first vector are introduced above. The fifth vector and the sixth vector can be determined in a similar manner and will not be elaborated here in detail.

[0198] Step 1150, based on the preset positional relationship, the fourth vector, the fifth vector, and the sixth vector, determine whether the first candidate position pair satisfies the second preset constraint condition, and obtain a second judgment result.

[0199] As Figure 13 shown, it is a flowchart of a method for determining whether a first candidate position pair satisfies a second preset constraint condition and obtaining a second judgment result based on a preset positional relationship, a fourth vector, a fifth vector, and a sixth vector provided by some exemplary embodiments of the present disclosure. Figure 13 The method shown may include Step 1310, Step 1320, Step 1330, and Step 1340.

[0200] Step 1310, determine a first angular attribute of the angle between the fourth vector and the fifth vector.

[0201] In some alternative embodiments of the present disclosure, a vector angle calculation method may be used to calculate the angle between the fourth vector and the fifth vector, and obtain the first angular attribute of the angle. The first angular attribute can be used to characterize whether the angle is an acute angle, a right angle, or an obtuse angle.

[0202] Step 1320, determine a second angular attribute of the angle between the fourth vector and the sixth vector.

[0203] In some alternative embodiments of the present disclosure, a vector angle calculation method may be used to calculate the angle between the fourth vector and the sixth vector, and obtain the second angular attribute of the angle. The second angular attribute can be used to characterize whether the angle is an acute angle, a right angle, or an obtuse angle.

[0204] Step 1330, in response to both the first angular attribute and the second angular attribute matching the preset positional relationship, obtain a second judgment result indicating that the first candidate position pair satisfies the second preset constraint condition.

[0205] Step 1340: In response to at least one of the first angle attribute and the second angle attribute not matching the preset positional relationship, obtain a second judgment result for characterizing that the first candidate position pair does not meet the second preset constraint condition.

[0206] In an alternative example, as Figure 14 shown, the first candidate position can be A b , the second candidate position can be B b , and the optical center position can be O b . The fourth vector can be represented as Z mb . The fifth vector pointing from the first candidate position to the optical center position can be represented as (AO) b . The sixth vector pointing from the second candidate position to the optical center position can be represented as (BO) b .

[0207] It should be noted that the reference object can be captured by the image sensor because light from the reference object enters the image sensor. The light from the reference object may be either the light generated by the reference object itself or the light in the environment reflected by the reference object. If the specified coordinate axis is in the same direction as the vector from the first surface to the second surface, then theoretically, the angle between (AO) b and Z mb should be an obtuse angle, and the angle between (BO) b and Z mb should also be an obtuse angle. If the specified coordinate axis is in the same direction as the vector from the second surface to the first surface, then theoretically, the angle between (AO) b and Z mb should be an acute angle, and the angle between (BO) b and Z mb should also be an acute angle.

[0208] For the case where the specified coordinate axis is in the same direction as the vector from the first surface to the second surface, if the first angle attribute is used to characterize that the angle between (AO) b and Z mb is indeed an obtuse angle, and the second angle attribute is used to characterize that the angle between (AO) b and Z mb is indeed an obtuse angle, this indicates that the theoretical situation is consistent with the actual situation, and both the first angle attribute and the second angle attribute match the preset positional relationship. Then, it can be determined that the first candidate position pair meets the second preset constraint condition, and a second judgment result for characterizing that the first candidate position pair meets the second preset constraint condition is obtained. If the first angle attribute is used to characterize that the angle between (AO) b and Z mb is an acute angle or a right angle, and / or the second angle attribute is used to characterize that the angle between (AO) b and Z mbis an acute angle or a right angle therebetween, which indicates that the theoretical situation does not match the actual situation, and at least one of the first angle attribute and the second angle attribute does not match the preset positional relationship. Then, it can be determined that the first candidate position pair does not meet the second preset constraint condition, and a second judgment result for representing that the first candidate position pair does not meet the second preset constraint condition is obtained.

[0209] For the case where the specified coordinate axis is in the same direction as the vector direction from the second surface to the first surface, if the first angle attribute is used to represent (AO) b and Z mb is indeed an acute angle therebetween, and the second angle attribute is used to represent (AO) b and Z mb is indeed an acute angle therebetween, which indicates that the theoretical situation matches the actual situation, and both the first angle attribute and the second angle attribute match the preset positional relationship. Then, it can be determined that the first candidate position pair meets the second preset constraint condition, and a second judgment result for representing that the first candidate position pair meets the second preset constraint condition is obtained. If the first angle attribute is used to represent (AO) b and Z mb is an acute angle or a right angle therebetween, and / or, the second angle attribute is used to represent (AO) b and Z mb is an acute angle or a right angle therebetween, which indicates that the theoretical situation does not match the actual situation, and at least one of the first angle attribute and the second angle attribute does not match the preset positional relationship. Then, it can be determined that the first candidate position pair does not meet the second preset constraint condition, and a second judgment result for representing that the first candidate position pair does not meet the second preset constraint condition is obtained.

[0210] It should be noted that the second preset constraint condition can be regarded as a constraint condition related to the light projection direction. Combining the relative pose and the optical center position, the rationality of the light projection direction (specifically the fifth vector and the sixth vector) related to the first candidate position pair can be verified, and based on this, the second judgment result can be obtained efficiently and reliably.

[0211] In the embodiments of the present disclosure, the first judgment result can be obtained by applying the first preset constraint condition, so as to provide a very effective reference for the outlier position screening process of the candidate position set. By applying the first preset constraint condition, a second vector that meets the requirements can also be found. The second vector can be used together with the first observed gravity direction, the second observed gravity direction, and the first vector to determine the relative pose between the first device and the second device. After determining the relative pose, the second judgment result can be obtained by applying the second preset constraint condition, so as to provide a very effective reference for the outlier position screening process of the candidate position set. Therefore, by adopting the embodiments of the present disclosure, the first preset constraint condition and the second preset constraint condition can be jointly applied, which is beneficial to ensuring the processing effect of the outlier position screening process of the candidate position set.

[0212] As Figure 15 shown, it is a schematic flowchart of a method for screening outlier positions provided by some other exemplary embodiments of the present disclosure. Figure 15 The method shown may include step 1510, step 1520, step 1530, and step 1540. Optionally, the combination of step 1510 to step 1540 may be an alternative implementation of step 250 of the present disclosure.

[0213] Step 1510, determine the relative pose between the first device and the second device; wherein, the relative pose is associated with the first observed gravity direction and the second observed gravity direction.

[0214] As introduced above, the second vector can be first searched in the target vector set, and then the relative pose can be determined by combining the first observed gravity direction, the second observed gravity direction, the first vector, and the second vector. Of course, the determination method of the relative pose is not limited to this. For example, the first device may be provided with a first magnetometer, and the second device may be provided with a second magnetometer. By combining the data collected by the first magnetometer and the second magnetometer respectively, and the first observed gravity direction and the second observed gravity direction, the relative pose can also be determined.

[0215] Step 1520, determine a first candidate position pair including a first candidate position and a second candidate position from the candidate position set.

[0216] Step 1530, based on the relative pose and the optical center position, determine whether the first candidate position pair meets the second preset constraint condition to obtain a second judgment result.

[0217] Step 1540, based on the second judgment result, perform outlier position screening processing on the candidate position set.

[0218] It should be noted that the specific implementation of step 1520 can refer to the relevant introduction of step 420 above, and the specific implementation of step 1530 and step 1540 can refer to the relevant introduction of step 750 and step 760 above, which will not be elaborated here.

[0219] Figure 15 In the illustrated embodiment, the first observed gravity direction and the second observed gravity direction can be used to determine the relative attitude. After determining the relative attitude, a second judgment result can be obtained by applying the second preset constraint condition, providing a very effective reference for the outlier position screening process of the candidate position set.

[0220] It can be seen that in the embodiments of the present disclosure, on the basis of using the first preset constraint condition, the second preset constraint condition can be used to implement the outlier position screening process of the candidate position set. Alternatively, it is not necessary to use the first preset constraint condition as the basis, but the second preset constraint condition can be used alone to implement the outlier position screening process of the candidate position set. In addition, the first preset constraint condition can also be used alone to implement the outlier position screening process of the candidate position set.

[0221] In some alternative embodiments of the present disclosure, the image sensor may include a monocular camera, and the image collected by the image sensor may include a monocular image.

[0222] As introduced above, in the embodiments of the present disclosure, based on the optical center position, the first observed gravity direction, the second observed gravity direction, and combined with different candidate positions in the same image, the outlier position screening process of the candidate position set can be effectively implemented. Therefore, in the embodiments of the present disclosure, only one camera can be set on the second device for image acquisition, which is beneficial to reducing the volume of the image sensor and minimizing the computational amount in the image processing process.

[0223] Any method for screening outlier positions provided by the embodiments of the present disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers, etc. Alternatively, any method for screening outlier positions provided by the embodiments of the present disclosure can be executed by a processor. For example, the processor executes any method for screening outlier positions mentioned in the embodiments of the present disclosure by calling the corresponding instructions stored in the memory. This will not be elaborated further below.

[0224] Exemplary device

[0225] As Figure 16 shown, it is a schematic structural diagram of a device for screening outlier positions provided by some exemplary embodiments of the present disclosure. Figure 16The device shown may include: a first determination module 161, configured to determine a first observed gravity direction through a first inertial measurement unit disposed on a first device; wherein, a reference object is disposed on the first device; a second determination module 163, configured to determine a second observed gravity direction through a second inertial measurement unit disposed on a second device; wherein, an image sensor is disposed on the second device; a third determination module 165, configured to determine a set of candidate positions of the reference object from an image collected by the image sensor; an acquisition module 167, configured to acquire the optical center position of the image sensor in a coordinate system corresponding to the second device; and a processing module 169, configured to perform an outlier position screening process on the set of candidate positions based on the first observed gravity direction, the second observed gravity direction, and the optical center position.

[0226] In some alternative embodiments of the present disclosure, two reference objects are disposed on the first device, as Figure 17 shown, the processing module 169 includes: a first determination sub-module 1691-A, configured to determine a first vector pointing from one of the two reference objects to the other in a coordinate system corresponding to the first device; a second determination sub-module 1693-A, configured to determine a first pair of candidate positions including a first candidate position and a second candidate position from the set of candidate positions; a first judgment sub-module 1695-A, configured to judge whether the first pair of candidate positions satisfies a first preset constraint condition based on the first observed gravity direction, the second observed gravity direction, the optical center position, and the first vector, to obtain a first judgment result; and a first processing sub-module 1697-A, configured to perform an outlier position screening process on the set of candidate positions based on the first judgment result.

[0227] In some alternative embodiments of the present disclosure, the first judgment sub-module 1695-A includes: a first acquisition unit, configured to obtain a first judgment result indicating that the first pair of candidate positions satisfies the first preset constraint condition in response to the existence of a second vector in a target vector set; and a second acquisition unit, configured to obtain a first judgment result indicating that the first pair of candidate positions does not satisfy the first preset constraint condition in response to the non-existence of the second vector in the target vector set; wherein, the target vector set includes: vectors with a starting point located on one of a first ray and a second ray and an end point located on the other of the first ray and the second ray, the first ray is a ray with a starting point at the optical center position and passing through the first candidate position, the second ray is a ray with a starting point at the optical center position and passing through the second candidate position, and the second vector is a vector with an included angle equal to a target angle with the second observed gravity direction, and the target angle is an included angle between the first vector and the first observed gravity direction.

[0228] In some alternative embodiments of the present disclosure, as Figure 18As shown in the figure, the device provided by the embodiment of the present disclosure further includes: a judgment module 181, configured to judge whether there is a solution for the independent variable x in the target equation; if the judgment result is yes, trigger the fourth determination module 183; if the judgment result is no, trigger the fifth determination module 185; the fourth determination module 183, configured to determine that there is a second vector in the target vector set in response to the fact that there is a solution for the independent variable x in the target equation; the fifth determination module 185, configured to determine that there is no second vector in the target vector set in response to the fact that there is no solution for the independent variable x in the target equation;

[0229] Among them, the target equation includes:

[0230] Ex 2 +Fx + H = 0, x ∈ (0, 1)

[0231] Among them,

[0232] E = [(b - a) · G b 2 -[(AB) m · G m 2 ||b - a|| 2

[0233] F = 2{(a · G b )[(b - a) · G b -[(AB) m · G m 2 [a · (b - a)]}

[0234] H = (a · G b ) 2 -[(AB) m · G m 2 ||a|| 2

[0235] a represents a unit vector in the same direction as the third vector, where the third vector refers to the vector pointing from the first candidate position to the optical center position, b represents a unit vector in the same direction as the fourth vector, where the fourth vector refers to the vector pointing from the optical center position to the second candidate position, (AB) m represents the first vector, G m represents the first observed gravity direction, G b represents the second observed gravity direction;

[0236] Among them, in response to the fact that there is a solution for the independent variable x in the target equation, then:

[0237] c = [a + x * (b - a)] / ||a + x * (b - a)||

[0238] c represents the second vector.​​​​

[0239] In some alternative embodiments of the present disclosure, as Figure 19 shown, the first processing sub-module 1697-A includes: a first determination unit 16971, configured to determine a relative attitude between the first device and the second device based on the first observed gravity direction, the second observed gravity direction, the first vector, and the second vector in response to the first determination result indicating that the first candidate position pair satisfies the first preset constraint condition; a first judgment unit 16973, configured to judge whether the first candidate position pair satisfies the second preset constraint condition based on the relative attitude and the optical center position, to obtain a second judgment result; and a first processing unit 16975, configured to perform an outlier position screening process on the candidate position set based on the second judgment result.

[0240] In some alternative embodiments of the present disclosure, the first determination unit 16971 includes: a first determination subunit, configured to determine a first cross product result of the first observed gravity direction and the first vector; a second determination subunit, configured to determine a second cross product result of the second observed gravity direction and the second vector; a first combination subunit, configured to combine the first observed gravity direction, the first vector, and the first cross product result to obtain a first combination matrix; a second combination subunit, configured to combine the second observed gravity direction, the second vector, and the second cross product result to obtain a second combination matrix; and a third determination subunit, configured to determine the relative attitude by using the inverse matrix of the first combination matrix and the second combination matrix.

[0241] In some alternative embodiments of the present disclosure, the first determination unit 16971 includes: a fourth determination subunit, configured to determine a first rotation matrix for rotating the first observed gravity direction to align with the second observed gravity direction; a fifth determination subunit, configured to determine a first unit vector in the same direction as the first component, where the first component refers to the component of the first vector perpendicular to the first observed gravity direction; a sixth determination subunit, configured to determine a second unit vector in the same direction as the second component, where the second component refers to the component of the second vector perpendicular to the second observed gravity direction; a rotation subunit, configured to rotate the first unit vector by using the first rotation matrix to obtain a third unit vector; a seventh determination subunit, configured to determine a second rotation matrix for rotating the third unit vector to align with the second unit vector; and an eighth determination subunit, configured to determine the relative attitude by using the first rotation matrix and the second rotation matrix.

[0242] In some alternative embodiments of the present disclosure, as Figure 20As shown, the first processing sub-module 1697-A includes: a second determination unit 16977, configured to, in response to the first determination result indicating that the first candidate position pair does not meet the first preset constraint condition, determine, from the candidate position set, a second candidate position pair including the first candidate position and the third candidate position; a second determination unit 16978, configured to determine whether the second candidate position pair meets the first preset constraint condition based on the first observed gravity direction, the second observed gravity direction, the optical center position, and the first vector, to obtain a third determination result; and a second processing unit 16979, configured to perform an outlier position screening process on the candidate position set based on the third determination result.

[0243] In some alternative embodiments of the present disclosure, as Figure 21 shown, the processing module 169 includes: a third determination sub-module 1691-B, configured to determine the relative pose between the first device and the second device; wherein the relative pose is associated with the first observed gravity direction and the second observed gravity direction; a fourth determination sub-module 1693-B, configured to determine, from the candidate position set, a first candidate position pair including the first candidate position and the second candidate position; a second determination sub-module 1695-B, configured to determine whether the first candidate position pair meets the second preset constraint condition based on the relative pose and the optical center position, to obtain a second determination result; and a second processing sub-module 1697-B, configured to perform an outlier position screening process on the candidate position set based on the second determination result.

[0244] In some alternative embodiments of the present disclosure, two reference objects are disposed on the first surface of the first device, and a specified coordinate axis of the coordinate system corresponding to the first device satisfies a preset positional relationship with the first surface; as Figure 22 shown, the second determination sub-module 1695-B includes: a third determination unit 16951, configured to determine a third vector for characterizing the specified coordinate axis; a conversion unit 16953, configured to use the relative pose to convert the third vector to the coordinate system corresponding to the second device, to obtain a fourth vector; a fourth determination unit 16955, configured to determine a fifth vector pointing from the first candidate position to the optical center position; a fifth determination unit 16957, configured to determine a sixth vector pointing from the second candidate position to the optical center position; and a third determination unit 16959, configured to determine whether the first candidate position pair meets the second preset constraint condition based on the preset positional relationship, the fourth vector, the fifth vector, and the sixth vector, to obtain a second determination result.

[0245] In some alternative embodiments of the present disclosure, the third determination unit 16959 includes: a ninth determination subunit, configured to determine a first angular property of an angle between a fourth vector and a fifth vector; a tenth determination subunit, configured to determine a second angular property of an angle between the fourth vector and a sixth vector; a first acquisition subunit, configured to obtain a second determination result for characterizing that a first candidate position pair satisfies a second preset constraint condition in response to both the first angular property and the second angular property matching a preset positional relationship; and a second acquisition subunit, configured to obtain a second determination result for characterizing that the first candidate position pair does not satisfy the second preset constraint condition in response to at least one of the first angular property and the second angular property not matching the preset positional relationship.

[0246] It should be noted that the specific structural composition of the second determination sub-module 1695-B is introduced above. The first determination unit 16973 in the above text can adopt a similar structural composition, which will not be elaborated here.

[0247] In some alternative embodiments of the present disclosure, the image sensor includes: a monocular camera, and the image collected by the image sensor includes: a monocular image.

[0248] In the device of the present disclosure, the various alternative embodiments, alternative implementation manners, and alternative examples disclosed above can be flexibly selected and combined as needed to achieve corresponding functions and effects, which will not be enumerated one by one in the present disclosure.

[0249] Exemplary electronic device

[0250] Figure 23 The block diagram of an electronic device according to an embodiment of the present disclosure is illustrated. The electronic device 2300 includes one or more processors 2310 and a memory 2320.

[0251] The processor 2310 may be a central processing unit (CPU) or other form of processing unit having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 2300 to perform desired functions.

[0252] The memory 2320 may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory, etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 2310 may run one or more computer program instructions to implement the methods of the various embodiments of the present disclosure described above and / or other desired functions.

[0253] In one example, the electronic device 2300 may further include: an input device 2330 and an output device 2340, and these components are interconnected through a bus system and / or other forms of connection mechanisms (not shown).

[0254] The input device 2330 may further include, for example, a keyboard, a mouse, etc.

[0255] The output device 2340 may output various information to the outside, which may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0256] Of course, for simplicity, Figure 23 only some of the components related to the present disclosure in the electronic device 2300 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application scenarios, the electronic device 2300 may further include any other appropriate components.

[0257] Exemplary computer program product and computer-readable storage medium

[0258] In addition to the above methods and devices, embodiments of the present disclosure may also be computer program products, which include computer program instructions that, when run by a processor, cause the processor to execute the steps in the methods according to various embodiments of the present disclosure described in the "Exemplary Methods" section above of this specification.

[0259] The computer program product may be written in any combination of one or more programming languages to write program code for performing the operations of the embodiments of the present disclosure. The programming languages include object-oriented programming languages, such as Java, C++, etc., and also include conventional procedural programming languages, such as the "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, executed as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0260] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, and when the computer program instructions are run by a processor, the processor is caused to execute the steps in the methods according to various embodiments of the present disclosure described in the "Exemplary Methods" section above of this specification.

[0261] The computer-readable storage medium may employ any combination of one or more readable media. The readable media may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0262] The basic principles of the present disclosure have been described above in connection with specific embodiments. However, the advantages, benefits, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, benefits, effects, etc. are essential for each embodiment of the present disclosure. The specific details disclosed above are only for the purposes of illustration and easy understanding, and are not limitations. The above details do not limit the present disclosure to necessarily implement using the above specific details.

[0263] Those skilled in the art can make various changes and modifications to the present disclosure without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present disclosure and their equivalent technologies, the present disclosure is also intended to include these changes and modifications.

Claims

1. A method for screening outlier positions, comprising: Determining a first observed gravity direction through a first inertial measurement unit disposed on a first device; wherein, a reference object is disposed on the first device; Determining a second observed gravity direction through a second inertial measurement unit disposed on a second device; wherein, an image sensor is disposed on the second device; Determining a set of candidate positions of the reference object from an image acquired by the image sensor; Obtaining the optical center position of the image sensor in the coordinate system corresponding to the second device; Performing outlier position screening processing on the set of candidate positions based on the first observed gravity direction, the second observed gravity direction, and the optical center position.

2. The method according to claim 1, wherein Two reference objects are disposed on the first device, and the performing outlier position screening processing on the set of candidate positions based on the first observed gravity direction, the second observed gravity direction, and the optical center position includes: Determining a first vector pointing from one of the two reference objects to the other in the coordinate system corresponding to the first device; Determining a first pair of candidate positions including a first candidate position and a second candidate position from the set of candidate positions; Judging whether the first pair of candidate positions satisfies a first preset constraint condition based on the first observed gravity direction, the second observed gravity direction, the optical center position, and the first vector, to obtain a first judgment result; Performing outlier position screening processing on the set of candidate positions based on the first judgment result.

3. The method according to claim 2, wherein, The judging whether the first pair of candidate positions satisfies a first preset constraint condition based on the first observed gravity direction, the second observed gravity direction, the optical center position, and the first vector, to obtain a first judgment result includes: Obtaining a first judgment result indicating that the first pair of candidate positions satisfies the first preset constraint condition in response to the existence of a second vector in a target vector set; Obtaining a first judgment result indicating that the first pair of candidate positions does not satisfy the first preset constraint condition in response to the non-existence of the second vector in the target vector set; Wherein, the target vector set includes: vectors with a starting point located on one of a first ray and a second ray and an end point located on the other of the first ray and the second ray, the first ray is a ray with a starting point at the optical center position and passing through the first candidate position, the second ray is a ray with a starting point at the optical center position and passing through the second candidate position, the second vector is a vector whose included angle with the second observed gravity direction is equal to a target angle, and the target angle is the included angle between the first vector and the first observed gravity direction.

4. The method according to claim 3, wherein The method further includes: Determining the existence of the second vector in the target vector set in response to the existence of a solution for the independent variable x in a target equation; Determining the non-existence of the second vector in the target vector set in response to the non-existence of a solution for the independent variable x in the target equation; Wherein, the target equation includes: Ex 2 +Fx + H = 0, x ∈ (0, 1) Wherein, E = [(b - a)·G b 2 - [(AB) m ·G m 2 ||b - a|| 2 ​​ F = 2{(a·G b )[(b - a)·G b - [(AB) m ·G m 2 [a·(b - a)]}​ H = (a·G b ) 2 −[(AB) m ·G m 2 ||a|| 2 ​ a represents a unit vector in the same direction as the third vector, where the third vector refers to the vector pointing from the first candidate position to the optical center position, and b represents a unit vector in the same direction as the fourth vector, where the fourth vector refers to the vector pointing from the optical center position to the second candidate position, (AB) m represents the first vector, G m represents the first observed gravity direction, G b represents the second observed gravity direction; Wherein, in response to the existence of a solution for the independent variable x in the target equation, then: c = [a + x*(b - a)] / ||a + x*(b - a)|| c represents the second vector.

5. The method according to claim 3, wherein Performing an outlier position screening process on the candidate position set based on the first judgment result includes: In response to the first judgment result indicating that the first candidate position pair satisfies the first preset constraint condition, determining the relative pose between the first device and the second device based on the first observed gravity direction, the second observed gravity direction, the first vector, and the second vector; Judging whether the first candidate position pair satisfies the second preset constraint condition based on the relative pose and the optical center position to obtain a second judgment result; Performing an outlier position screening process on the candidate position set based on the second judgment result.

6. The method according to claim 5, wherein, Determining the relative pose between the first device and the second device based on the first observed gravity direction, the second observed gravity direction, the first vector, and the second vector includes: Determining a first cross product result of the first observed gravity direction and the first vector; Determining a second cross product result of the second observed gravity direction and the second vector; Combining the first observed gravity direction, the first vector, and the first cross product result to obtain a first combined matrix; Combining the second observed gravity direction, the second vector, and the second cross product result to obtain a second combined matrix; Determining the relative pose using the inverse matrix of the first combined matrix and the second combined matrix.

7. The method according to claim 5, wherein Determining the relative pose between the first device and the second device based on the first observed gravity direction, the second observed gravity direction, the first vector, and the second vector includes: Determining a first rotation matrix for rotating the first observed gravity direction to align with the second observed gravity direction; Determining a first unit vector in the same direction as the first component; where the first component refers to the component of the first vector perpendicular to the first observed gravity direction; Determining a second unit vector in the same direction as the second component; where the second component refers to the component of the second vector perpendicular to the second observed gravity direction; Rotating the first unit vector using the first rotation matrix to obtain a third unit vector; Determining a second rotation matrix for rotating the third unit vector to align with the second unit vector; Determining the relative pose using the first rotation matrix and the second rotation matrix.

8. The method according to claim 3, wherein Performing an outlier position screening process on the candidate position set based on the first judgment result includes: In response to the first judgment result indicating that the first candidate position pair does not satisfy the first preset constraint condition, determining a second candidate position pair including the first candidate position and a third candidate position from the candidate position set; Judging whether the second candidate position pair satisfies the first preset constraint condition based on the first observed gravity direction, the second observed gravity direction, the optical center position, and the first vector to obtain a third judgment result; Performing an outlier position screening process on the candidate position set based on the third judgment result.

9. The method according to claim 1, wherein, Performing an outlier position screening process on the candidate position set based on the first observed gravity direction, the second observed gravity direction, and the optical center position includes: Determining a relative pose between the first device and the second device; wherein, the relative pose is associated with the first observed gravity direction and the second observed gravity direction; Determining, from the candidate position set, a first candidate position pair including a first candidate position and a second candidate position; Judging whether the first candidate position pair satisfies a second preset constraint condition based on the relative pose and the optical center position, to obtain a second judgment result; Performing an outlier position screening process on the candidate position set based on the second judgment result.

10. The method according to claim 5 or 9, wherein, The two reference objects are disposed on a first surface of the first device, and a specified coordinate axis of the coordinate system corresponding to the first device and the first surface satisfy a preset positional relationship; The judging whether the first candidate position pair satisfies a second preset constraint condition based on the relative pose and the optical center position, to obtain a second judgment result, includes: Determining a third vector for characterizing the specified coordinate axis; Using the relative pose to transform the third vector to the coordinate system corresponding to the second device, to obtain a fourth vector; Determining a fifth vector pointing from the first candidate position to the optical center position; Determining a sixth vector pointing from the second candidate position to the optical center position; Judging whether the first candidate position pair satisfies a second preset constraint condition based on the preset positional relationship, the fourth vector, the fifth vector, and the sixth vector, to obtain a second judgment result.

11. The method according to claim 10, wherein, The judging whether the first candidate position pair satisfies a second preset constraint condition based on the preset positional relationship, the fourth vector, the fifth vector, and the sixth vector, to obtain a second judgment result, includes: Determining a first angular attribute of an angle between the fourth vector and the fifth vector; Determining a second angular attribute of an angle between the fourth vector and the sixth vector; In response to both the first angular attribute and the second angular attribute matching the preset positional relationship, obtaining a second judgment result for characterizing that the first candidate position pair satisfies the second preset constraint condition; In response to at least one of the first angular attribute and the second angular attribute not matching the preset positional relationship, obtaining a second judgment result for characterizing that the first candidate position pair does not satisfy the second preset constraint condition.

12. According to the method described in any one of claims 1-9 and 11, wherein The image sensor includes: a monocular camera, and the image collected by the image sensor includes: a monocular image.

13. A device for screening outlier positions, including: A first determination module, configured to determine a first observed gravity direction through a first inertial measurement unit disposed on a first device; wherein, a reference object is disposed on the first device; A second determination module, configured to determine a second observed gravity direction through a second inertial measurement unit disposed on a second device; wherein, an image sensor is disposed on the second device; A third determination module, configured to determine a candidate position set of the reference object from the image collected by the image sensor; An acquisition module, configured to acquire the position of the optical center of the image sensor in the coordinate system corresponding to the second device; A processing module, configured to perform an outlier position screening process on the candidate position set based on the first observed gravity direction, the second observed gravity direction, and the position of the optical center.

14. An electronic device, comprising: A memory, configured to store a computer program product; A processor, configured to execute the computer program product stored in the memory, and when the computer program product is executed, implement the method for screening outlier positions according to any one of claims 1 to 12 above.

15. A computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, implement the method for screening outlier positions according to any one of claims 1 to 12 above.

Images