A global positioning method, device, movable device and electronic device

Abstract

Embodiments of the present application provide a global positioning method and device, a movable device and an electronic device. The method comprises the following steps: first, extracting first geometric features of a first image and second geometric features of a second image, which are captured in different poses; then, projecting the first geometric features and the second geometric features into the same coordinate system. Since the projection is based on the height of the ceiling, only the second geometric features with the actual height of the ceiling will coincide with the first geometric features during the projection. The coinciding target first geometric features are the features on the ceiling, and features not belonging to the ceiling can be quickly filtered out. Finally, the target pose of the movable device when the first image is captured is determined based on the target geometric features of the ceiling. Since the features not belonging to the ceiling can be quickly filtered out according to the projection deviation of the features of the images captured in different poses, complex calculations are not required, and the efficiency of global positioning is improved.

Description

Technical Field

[0001] This application relates to the field of image positioning technology, and in particular to a global positioning method, apparatus, mobile device, and electronic device. Background Technology

[0002] In existing technologies, some applications (such as robotic vacuum cleaners) use ceiling features for global localization. Global localization using ceiling features typically employs a top-view visual SLAM (Simultaneous Localization and Mapping) scheme. However, when the robot captures images of the ceiling, there are often many obstructions, which affect the final localization result. Current technologies for removing these obstructions rely on deep learning methods, which consume significant computational resources, resulting in low localization efficiency. Summary of the Invention

[0003] The purpose of this application is to provide a global positioning method, apparatus, mobile device, and electronic device to quickly remove obstructions and improve global positioning efficiency. The specific technical solution is as follows:

[0004] In a first aspect of this application, a global positioning method is provided, the method comprising:

[0005] Acquire a first image and a second image of the ceiling taken by the mobile device in different poses, and the relative pose between the poses of the mobile device when it takes the first image and the second image;

[0006] Extract the first geometric features of the first image and the second geometric features of the second image;

[0007] Based on the relative pose and the camera intrinsics of the mobile device, the first geometric feature and the second geometric feature are projected onto the same coordinate system with the height of the ceiling as the height of the first geometric feature and the second geometric feature, to obtain a feature projection image.

[0008] On the feature projection image, a first geometric feature that coincides with the second geometric feature is determined as the target first geometric feature;

[0009] Based on the first geometric features of the target, the target pose when the mobile device captures the first image is determined.

[0010] In one possible implementation, the step of projecting the first and second geometric features onto the same coordinate system based on the relative pose and the camera intrinsics of the mobile device, using the ceiling height as the height of the first and second geometric features, to obtain a feature projection image includes:

[0011] Based on the relative pose and the camera intrinsics of the mobile device, establish a conversion relationship between the first image on the ceiling and the second image on the ceiling;

[0012] Based on the transformation relationship, the second geometric feature is projected onto the first image to obtain a feature projection image.

[0013] In one possible implementation, the geometric features include line features;

[0014] The extraction of the first geometric features of the first image and the second geometric features of the second image includes:

[0015] Line features with an angle greater than a preset angle threshold to the direction of motion are extracted from the first image and used as first geometric features; and line features with an angle greater than the preset angle threshold to the direction of motion are extracted from the second image and used as second geometric features; wherein, the direction of motion is the direction of movement of the mobile device from capturing the second image to capturing the first image.

[0016] In one possible implementation, the method further includes:

[0017] Extract line features that are perpendicular to and connected to the first geometric feature of the target from the first image, as supplementary first geometric features;

[0018] Determining the target pose of the mobile device when it captures the first image based on the first geometric features of the target includes:

[0019] Based on the target's first geometric features and the supplementary first geometric features, the target pose when the mobile device captures the first image is determined.

[0020] In one possible implementation, determining a first geometric feature that coincides with the second geometric feature on the feature projection image, as the target first geometric feature, includes:

[0021] On the feature projection image, a first geometric feature with an overlap degree greater than a preset overlap threshold is determined as the target first geometric feature.

[0022] In a second aspect of this application, a mobile device is provided, the mobile device comprising:

[0023] A pose sensor is used to monitor the pose of the mobile device in real time.

[0024] A camera for photographing the ceiling when the mobile device is in different poses;

[0025] The processor is configured to acquire a first image and a second image of the ceiling captured by the camera; extract a first geometric feature from the first image and a second geometric feature from the second image; acquire a relative pose between a first pose sensed by the pose sensor when the mobile device captures the first image and a second pose sensed by the pose sensor when the mobile device captures the second image; based on the relative pose and the camera intrinsic parameters of the mobile device, project the first geometric feature and the second geometric feature onto the same coordinate system with the height of the ceiling as the height of the first geometric feature and the second geometric feature to obtain a feature projection image; determine a first geometric feature that coincides with the second geometric feature on the feature projection image as a target first geometric feature; and determine the target pose of the mobile device when it captures the first image based on the target first geometric feature.

[0026] In one possible implementation, the processor, based on the relative pose and the camera intrinsics of the mobile device, projects the first and second geometric features onto the same coordinate system, using the ceiling height as the height of the first and second geometric features, to obtain a feature projection image, including:

[0027] Based on the relative pose and the camera intrinsics of the mobile device, establish a conversion relationship between the first image on the ceiling and the second image on the ceiling;

[0028] Based on the transformation relationship, the second geometric feature is projected onto the first image to obtain a feature projection image;

[0029] The geometric features include line features;

[0030] The extraction of the first geometric features of the first image and the second geometric features of the second image includes:

[0031] Line features with an angle greater than a preset angle threshold to the direction of motion are extracted from the first image and used as first geometric features; and line features with an angle greater than the preset angle threshold to the direction of motion are extracted from the second image and used as second geometric features; wherein, the direction of motion is the direction of movement of the mobile device from capturing the second image to capturing the first image.

[0032] The processor is further configured to extract line features that are perpendicular to and connected to the first geometric feature of the target in the first image, as supplementary first geometric features;

[0033] Determining the target pose of the mobile device when it captures the first image based on the first geometric features of the target includes:

[0034] Based on the target's first geometric features and the supplementary first geometric features, the target pose when the mobile device captures the first image is determined.

[0035] The step of determining the first geometric feature that coincides with the second geometric feature on the feature projection image as the target first geometric feature includes:

[0036] On the feature projection image, a first geometric feature with an overlap degree greater than a preset overlap threshold with the second geometric feature is determined and used as the target first geometric feature.

[0037] In a third aspect of this application, a global positioning device is provided, the device comprising:

[0038] The acquisition module is used to acquire a first image and a second image of the ceiling taken by the mobile device in different poses, as well as the relative pose between the poses of the mobile device when it takes the first image and the second image.

[0039] The extraction module is used to extract the first geometric features of the first image and the second geometric features of the second image;

[0040] The projection module is used to project the first geometric feature and the second geometric feature onto the same coordinate system based on the relative pose and the camera intrinsic parameters of the mobile device, with the height of the ceiling as the height of the first geometric feature and the second geometric feature, to obtain a feature projection image.

[0041] The target feature determination module is used to determine a first geometric feature that coincides with the second geometric feature on the feature projection image, as the target first geometric feature;

[0042] The target pose determination module is used to determine the target pose when the mobile device captures the first image based on the first geometric features of the target.

[0043] In one possible implementation, the projection module, based on the relative pose and the camera intrinsics of the mobile device, projects the first and second geometric features onto the same coordinate system, using the ceiling height as the height of the first and second geometric features, to obtain a feature projection image, including:

[0044] Based on the relative pose and the camera intrinsics of the mobile device, establish a conversion relationship between the first image on the ceiling and the second image on the ceiling;

[0045] Based on the transformation relationship, the second geometric feature is projected onto the first image to obtain a feature projection image.

[0046] In one possible implementation, the geometric features include line features;

[0047] The extraction module extracts the first geometric features of the first image and the second geometric features of the second image, including:

[0048] Line features with an angle greater than a preset angle threshold to the direction of motion are extracted from the first image and used as first geometric features; and line features with an angle greater than the preset angle threshold to the direction of motion are extracted from the second image and used as second geometric features; wherein, the direction of motion is the direction of movement of the mobile device from capturing the second image to capturing the first image.

[0049] The device further includes:

[0050] The supplementary feature determination module is used to extract line features that are perpendicular to and connected to the target first geometric feature in the first image, as supplementary first geometric features;

[0051] Determining the target pose of the mobile device when it captures the first image based on the first geometric features of the target includes:

[0052] Based on the target's first geometric features and the supplementary first geometric features, the target pose when the mobile device captures the first image is determined.

[0053] The target feature determination module determines a first geometric feature that coincides with the second geometric feature on the feature projection image, as the target first geometric feature, including:

[0054] On the feature projection image, a first geometric feature with an overlap degree greater than a preset overlap threshold is determined as the target first geometric feature.

[0055] In a fourth aspect of this application, an electronic device is provided, comprising:

[0056] Memory, used to store computer programs;

[0057] When a processor executes a program stored in memory, it implements any of the methods described in the first aspect above.

[0058] In a fifth aspect of this application, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when executed by a processor, the computer program implements any of the methods described in the first aspect above.

[0059] Beneficial effects of the embodiments in this application:

[0060] This application provides a global positioning method, apparatus, and electronic device that can acquire first and second images of a ceiling captured by a mobile device in different poses, as well as the relative pose between the poses of the mobile device when capturing the first and second images. It extracts first geometric features from the first image and second geometric features from the second image. Based on the relative pose and camera intrinsics of the mobile device, and using the height of the ceiling as the height of the first and second geometric features, it projects the first and second geometric features onto the same coordinate system to obtain a feature projection image. It then determines a target first geometric feature that overlaps with the second geometric feature on the feature projection image. Finally, it determines the target pose of the mobile device when capturing the first image based on the target first geometric feature. In this application embodiment, by first extracting the first geometric features of the first image captured in different poses and the second geometric features of the second image, and then projecting the first and second geometric features onto the same coordinate system, since the projection is based on the height of the ceiling, only the second geometric features whose actual height is the ceiling will overlap with the first geometric features during projection. These overlapping target first geometric features are the features on the ceiling, thus quickly filtering out features that do not belong to the ceiling. Finally, the target pose of the mobile device when the first image was captured was determined based on the target's geometric features on the ceiling. Since features not belonging to the ceiling can be quickly filtered out simply by considering the feature projection deviations of images captured at different poses, complex calculations are not required, thus improving the efficiency of global localization.

[0061] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0063] Figure 1 This is a schematic flowchart of a global positioning method provided in an embodiment of this application;

[0064] Figure 2-1This application provides a schematic diagram illustrating the extraction of line features from a top-view image 1 to characterize the top-view image 1.

[0065] Figure 2-2 A schematic diagram illustrating the process of extracting a first geometric feature from a first image and a second geometric feature from a second image, provided in an embodiment of this application;

[0066] Figure 2-3 A camera pinhole imaging model provided in this application embodiment;

[0067] Figure 2-4 A schematic diagram illustrating the projection of a second geometric feature onto a first image, provided as an embodiment of this application;

[0068] Figure 2-5 A schematic diagram of a ceiling being photographed by a mobile device, as provided in an embodiment of this application;

[0069] Figure 2-6 A supplementary first geometric feature schematic diagram provided for an embodiment of this application;

[0070] Figure 3 A schematic diagram of a mobile device provided in an embodiment of this application;

[0071] Figure 4 A schematic diagram of a global positioning device provided in an embodiment of this application;

[0072] Figure 5 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0073] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0074] This application proposes a global localization method that can quickly remove features of occluded areas from images obtained by photographing ceilings, thereby enabling rapid global localization.

[0075] This application proposes a global positioning method, such as... Figure 1 As shown, the method includes:

[0076] S101. Acquire a first image and a second image of the ceiling taken by the mobile device in different poses, and the relative pose between the poses of the mobile device when it takes the first image and the second image.

[0077] S102, Extract the first geometric features of the first image and the second geometric features of the second image.

[0078] S103. Based on the relative pose and the camera intrinsics of the mobile device, using the height of the ceiling as the height of the first geometric feature and the second geometric feature, project the first geometric feature and the second geometric feature onto the same coordinate system to obtain the feature projection image.

[0079] S104. Determine the first geometric feature that coincides with the second geometric feature on the feature projection image, and use it as the target first geometric feature.

[0080] S105. Based on the first geometric features of the target, determine the target pose when the mobile device captures the first image.

[0081] In this embodiment, the first geometric features of the first image and the second geometric features of the second image are extracted from the first image captured by the mobile device in different poses. Then, the first and second geometric features are projected onto the same coordinate system. Since both the first and second images are captured from the ceiling, and the geometric features of the ceiling (and obstacles below the ceiling and above the mobile device) are fixed, the first and second geometric features can be considered as projections of geometric features of the same field onto different coordinate systems. Therefore, if the first and second geometric features are projected onto the same coordinate system in an accurate manner, the first geometric feature will coincide with the second geometric feature. However, since the projection is based on the height of the ceiling, only geometric features whose actual height is the height of the ceiling will be projected in an accurate manner, while geometric features whose actual height is not the height of the ceiling will be projected in an incorrect manner. Therefore, only the second geometric features whose actual height is the height of the ceiling will coincide with the first geometric features. Conversely, these overlapping target first geometric features are features on the ceiling, thus allowing for the rapid filtering out of features that do not belong to the ceiling. The geometric features not belonging to the ceiling are clearly the geometric features of obstacles. Therefore, determining the target pose when the mobile device captures the first image based on the target's first geometric features of the ceiling is equivalent to eliminating the interference of obstacle geometric features on global localization, effectively improving the accuracy of global localization. Furthermore, since the global localization provided by this application can quickly filter out features not belonging to the ceiling simply by using the feature projection deviation of images captured at different poses, complex calculations are not required, thus improving the efficiency of global localization.

[0082] The above S101-S105 will be described in detail below.

[0083] In S101, the mobile device is any intelligent device with shooting capabilities and autonomous movement, such as a smart robotic vacuum cleaner or an automated guided vehicle. The camera on the mobile device moves with the mobile device. The camera is fixed to the mobile device. During shooting, if the mobile device's pose remains constant, the image captured by the same camera pose is the same. If the mobile device's pose remains constant, the images captured by different camera poses are different. If the mobile device's pose changes, the image captured by the same camera pose is different, and the image captured by different camera poses is also different. In other words, the captured image changes with the mobile device's pose, and the camera pose also affects the captured image. Therefore, in this embodiment, global localization is performed based on the image captured by the camera and the mobile device's pose; that is, global localization is performed by combining the camera pose and the mobile device's pose.

[0084] Relative pose is the relative positional relationship between the pose when the first image is captured and the pose when the second image is captured. Assuming the pose when the first image is captured is the first pose, and the pose when the second image is captured is the second pose, the relative pose can be determined by the process of moving from the second pose to the first pose. For example, an IMU can be used to acquire motion parameters during the movement from the second pose to the first pose to calculate the relative pose. Alternatively, the first and second poses can be acquired using any existing method for obtaining poses, and then the relative pose can be determined based on the first and second poses. However, the first and second poses here have relatively low accuracy. Furthermore, the first pose follows the second pose; that is, the time when the second image is captured precedes the time when the first image is captured. If capturing the first image represents the current pose, then capturing the second image represents a historical pose of the current pose; conversely, if capturing the first image represents a first historical pose, then capturing the second image represents a historical pose of the first historical pose. In this embodiment, global positioning refers to locating the pose of the mobile device when the first image is captured. It is not limited to global positioning of the current pose of the mobile device, but can also be global positioning of the historical pose of the mobile device.

[0085] In S102, the geometric features of the image are those that can characterize the geometric structure of the image. These can be line features (i.e., lines that can characterize the image) or point features (i.e., points that can characterize the image), etc. The number of first geometric features can be one or more, and the same applies to second geometric features. For ease of description, the following explanation will only focus on the case where there are multiple first geometric features and multiple second geometric features. The principle is the same for the case where there is only one first geometric feature and one second geometric feature, and will not be elaborated here.

[0086] For example, see Figure 2-1 ,like Figure 2-1 The diagram shows the extraction of line features from top view image 1 to characterize the features of top view image 1.

[0087] In S103, projecting the first geometric feature and the second geometric feature onto the same coordinate system can include the following three cases:

[0088] Case 1: Project the first geometric feature onto the image coordinate system of the second image;

[0089] Scenario 2: Project the second geometric feature onto the image coordinate system of the first image;

[0090] Case 3: Project both the first and second geometric features into the third image coordinate system.

[0091] The third image coordinate system is any image coordinate system other than the image coordinate systems of the first and second images.

[0092] It is understandable that the coordinate transformation relationship between different image coordinate systems can be determined based on the camera's imaging principle. For ease of description, the following text assumes that the camera operates based on the pinhole imaging principle, then... Figure 2-3 The diagram shows a pinhole camera imaging model, which includes point P, imaging point P', optical center O, focal length f, pixel plane, and physical imaging plane. The plane containing the optical center is the xoy plane, and the plane containing the pixel plane and the physical imaging plane is the xoy plane. ’ 0y ’ Plane. In the camera coordinate system, there exists an arbitrary point P = [X, Y, Z]. On the physical imaging plane, there exists a point P' = [u, v] corresponding to point P. During the projection process, the X and Y axes of the coordinate systems of the two points are consistent. Therefore, the image P' can have an additional dimension f, forming coordinates P' = [u, v, f] in the camera coordinate system, where f is the focal length of the camera and c is the image pixel value.

[0093] The transformation relationship from point P to point P' can be determined using the following formula:

[0094]

[0095] Where Z is the ceiling height, K is the camera's intrinsic parameter, and f x It is the X-axis component of the camera's focal length in the camera coordinate system, f y It is the y-component of the camera's focal length in the camera coordinate system, c x It is the width in pixels of the physical imaging plane image, c y It is the height pixel value of the physical imaging plane image.

[0096] This application is based on the principle of pinhole imaging. According to the relative pose and camera intrinsic parameters of the mobile device, the height of the ceiling is used as the height of the first and second geometric features. The first and second geometric features are projected onto the same coordinate system to obtain a feature projection image. Since obtaining the feature projection image is not the focus of this application, and it can be obtained using existing technology, it will not be elaborated upon further in this embodiment.

[0097] In S104, when the second geometric feature is projected onto the first image, the resulting feature projection image includes the projection features of both the first and second geometric features. For example, according to Case 1, the feature projection image includes: the projection features of the first geometric feature and the second geometric feature; according to Case 2, the feature projection image includes: the projection features of the second geometric feature and the first geometric feature; according to Case 3, the feature projection image includes: the projection features of the first geometric feature and the projection features of the second geometric feature. Figure 2-4 As shown, the projection feature is composed of countless projection points; the projection feature of the geometric feature is used to characterize the geometric feature. Therefore, in this application, the projection feature of the geometric feature and the geometric feature are not distinguished.

[0098] The ceiling height is the distance from the position of the mobile device's camera to the top of the ceiling. For example... Figure 2-5 In the diagram, Z represents the height of the ceiling.

[0099] Because, during the projection process, due to various objective reasons, the first and second geometric features in the feature projection image cannot be 100% identical, in order to avoid some geometric features that should be identical being misjudged as non-identical due to objective reasons, the above-mentioned S104 can specifically be as follows:

[0100] S1041. On the feature projection image, determine the first geometric feature whose overlap with the first geometric feature is greater than a preset overlap threshold, and use it as the target first geometric feature.

[0101] In this step, the preset overlap threshold is determined by those skilled in the art based on the actual application scenario. For example, assuming the preset overlap threshold is 85%, the first geometric feature on the feature projection image with an overlap of more than 85% with the first geometric feature is taken as the target first geometric feature. In this embodiment, the larger the preset overlap threshold, the lower the probability of determining the set features of obstacles as the first target set features; therefore, the global positioning is less affected by obstacles, and the probability of not determining the geometric features on the ceiling as the first target set features is lower, thus making the geometric features based on global positioning more comprehensive.

[0102] In this embodiment, a preset overlap threshold is used to determine the first geometric feature of the target. This can avoid some geometric features that should overlap being misjudged as non-overlapping geometric features due to objective reasons, thereby improving the accuracy of determining the first geometric feature of the target.

[0103] In S105, since both the first and second images are obtained by photographing the ceiling, and the geometric features of the ceiling (and obstacles below the ceiling and above the mobile device) are fixed, the first and second geometric features can be considered as projections of geometric features of the same field in different coordinate systems. Therefore, if the first and second geometric features are projected onto the same coordinate system in an accurate projection method, the first geometric feature will coincide with the second geometric feature. However, since the projection is based on the height of the ceiling, only geometric features whose actual height is the height of the ceiling will be projected in an accurate projection method, while geometric features whose actual height is not the height of the ceiling will be projected in an incorrect projection method. Therefore, only the second geometric feature whose actual height is the height of the ceiling will coincide with the first geometric feature. Conversely, these overlapping target first geometric features are the features on the ceiling.

[0104] Since the first geometric feature of the target must be a geometric feature on the ceiling, and each feature on the ceiling can be located to a pose, the target pose when the mobile device captures the first image can be determined by the first geometric feature of the target.

[0105] In this embodiment, the purpose of global localization is to determine the pose of the mobile device when the first image is captured. When projecting according to cases one and three, the resulting feature projection image is in the second or third image coordinate system. During localization, the obtained target first geometric features need to be projected back onto the first image coordinate system to determine the target pose of the mobile device when the first image is captured. However, when projecting according to case two, the resulting feature projection image is in the first image coordinate system. Therefore, the target first geometric features can be directly used to determine the target pose of the mobile device when the first image is captured. Therefore, to improve computational efficiency, the second geometric features are projected onto the first image, eliminating the need to project the target first geometric features again, thus reducing a large number of projection steps and improving the efficiency of global localization.

[0106] Based on this, the above S103 specifically includes:

[0107] S1031. Based on the relative pose and the camera intrinsics of the mobile device, establish the transformation relationship between the first image on the ceiling and the second image on the ceiling.

[0108] In this step, projection can be performed using the following formula:

[0109]

[0110] Where Z is the height of the ceiling, K is the camera intrinsic parameter matrix, and P uv These are the coordinates of the projected image, P. w R is the coordinate of the projected image, t is the rotation matrix, T is the pose change matrix, and u and v are geometric eigenvalues.

[0111] Projecting the second geometric feature onto the first image, then P uv P is the coordinate of the first geometric feature of the first image. w Let be the coordinates of the second geometric feature, and u and v be the eigenvalues ​​of the first geometric feature. R is the rotation matrix during the transformation from the second geometric feature to the first geometric feature, t is the displacement of the second geometric feature to the first geometric feature, and T is the pose transformation matrix of the second geometric feature to the first geometric feature.

[0112] The definite relationship can be determined using the formula above.

[0113] S1032. According to the transformation relationship, the second geometric feature is projected onto the first image to obtain the feature projection image.

[0114] In this step, the obtained feature projection image includes a first geometric feature and a second geometric feature.

[0115] In this embodiment, the second geometric feature is projected onto the first image. Therefore, when determining the first feature of the target, there is no need to project the first geometric feature again, thereby reducing a large number of projection steps and improving the efficiency of global positioning.

[0116] Since there are no complex patterns on the ceiling, and the ceiling texture is composed of multiple intersecting line segments, and line features require much less computation than point features, in one possible embodiment, the aforementioned geometric features include line features.

[0117] However, line features in the same direction of motion are inaccurate. Therefore, to further improve the efficiency of global localization, S102 may specifically include:

[0118] S1021. Extract line features in the first image whose angle with the direction of motion is greater than a preset angle threshold, as the first geometric feature; and extract line features in the second image whose angle with the direction of motion is greater than a preset angle threshold, as the second geometric feature; wherein, the direction of motion is the direction of movement of the mobile device from capturing the second image to capturing the first image.

[0119] In this step, some line features in the first image are inaccurate. To improve the efficiency of global positioning, these inaccurate line features need to be pre-filtered out. In this embodiment, line features smaller than a preset angle threshold are these interfering line features. The preset angle threshold is determined by those skilled in the art based on actual application scenarios and is not limited here.

[0120] In this embodiment, geometric features include line features. Interfering line features are filtered out by using a preset included angle threshold. Compared with point features or other features, line features require less computation and are more accurate, thereby improving the positioning efficiency of global positioning.

[0121] In one possible embodiment, when the geometric features include line features, some inaccurate line features are filtered out. However, during subsequent localization, some features related to the target's first geometric features are also filtered out. The target's first geometric features are not comprehensive enough, and the accuracy of global localization based solely on them is low. To improve the accuracy of global localization, the above method further includes:

[0122] S106. Extract line features that are perpendicular to and connected to the first geometric feature of the target from the first image, as supplementary first geometric features.

[0123] In this step, the features in the supplementary first geometric feature and the features in the target first geometric feature do not overlap. That is, the target first geometric feature cannot be the supplementary first geometric feature. The supplementary first geometric feature is a feature that complements the target first geometric feature. For example... Figure 2-6 As shown, the determination of the supplementary first geometric feature is based on the existing line features in the first image. That is, the supplementary first geometric feature is the line feature existing in the first image.

[0124] Specifically, S105 may include:

[0125] S1051. Based on the first geometric features of the target and the supplementary first geometric features, determine the target pose when the mobile device captures the first image.

[0126] In this embodiment, supplementary first geometric features are used to replenish some line features related to the target's first geometric features that were filtered out from the target's first geometric features. Therefore, the features are more comprehensive when performing global positioning, thereby improving the accuracy of global positioning.

[0127] This application also provides a global positioning method, the method comprising:

[0128] S201. Obtain a first image and a second image of the ceiling taken by the mobile device in different poses, and the relative pose between the poses of the mobile device when it takes the first image and the second image.

[0129] This step is the same as S101 above, and will not be described in detail here.

[0130] S202. Extract line features from the first image whose angle with the direction of motion is greater than a preset angle threshold, as the first geometric feature; and extract line features from the second image whose angle with the direction of motion is greater than a preset angle threshold, as the second geometric feature; the direction of motion is the movement direction of the mobile device from capturing the second image to capturing the first image.

[0131] This step is the same as S1021 mentioned above, and will not be described in detail here. Figure 2-2 The diagram illustrates the process of extracting the first geometric features from the first image and the second geometric features from the second image. The first image is a top-view image 1, and the second image is a top-view image 2. Top-view image 1 is captured by the mobile device in pose 1, and top-view image 2 is captured by the mobile device in pose 2. The ceiling height is the distance from the top of the mobile device where the top-view image is captured to the top of the ceiling.

[0132] S203. Based on the relative pose and the camera intrinsics of the mobile device, establish the transformation relationship between the first image on the ceiling and the second image on the ceiling.

[0133] This step is the same as S1031 mentioned above, and will not be described in detail here.

[0134] S204. Based on the transformation relationship, project the second geometric feature onto the first image to obtain the feature projection image.

[0135] This step is the same as S1032 mentioned above, and will not be described in detail here.

[0136] S205. Determine the first geometric feature that coincides with the second geometric feature on the feature projection image, and use it as the target first geometric feature.

[0137] This step is the same as S104 above, and will not be described in detail here.

[0138] S206. Extract line features that are perpendicular to and connected to the first geometric feature of the target in the first image, as supplementary first geometric features.

[0139] This step is the same as S106 mentioned above, and will not be elaborated on further here.

[0140] S207. Based on the first geometric features of the target and the supplementary first geometric features, determine the target pose when the mobile device captures the first image.

[0141] This step is the same as S1051 mentioned above, and will not be described in detail here.

[0142] In this embodiment, firstly, the first geometric features of the first image captured from different poses and the second geometric features of the second image are extracted. Then, the second geometric features are projected onto the first image. Since the projection is based on the ceiling height, only second geometric features whose actual height is the ceiling will overlap with the first geometric features in the first image. These overlapping target first geometric features are the features on the ceiling, thus quickly filtering out features that do not belong to the ceiling and identifying the target first geometric features. Then, the line features related to the target first geometric features are added back. Finally, based on the target first geometric features of the ceiling and the supplemented first geometric features, the target pose of the mobile device when capturing the first image is determined. Since features that do not belong to the ceiling can be quickly filtered out simply by the feature projection deviation of the images captured from different poses, complex calculations are not required, thus improving the efficiency of global positioning.

[0143] This application also provides a mobile device, such as... Figure 3 As shown, the mobile device includes:

[0144] The pose sensor 301 is used to monitor the pose of the mobile device in real time.

[0145] When the pose sensor 301 monitors the pose of the mobile device in real time, it saves the pose information so that the subsequent processor 303 can directly obtain the first pose and the second pose from the pose sensor 301.

[0146] Camera 302 is used to photograph the ceiling when the mobile device is in different poses;

[0147] Camera 302 is mounted on top of a mobile device, and its shooting angle is movable. Therefore, when capturing an image of the ceiling, a camera pose exists. The camera pose characterizes the shooting angle of the camera when capturing the image. The images captured by camera 302 include a first image and a second image.

[0148] Processor 303 is configured to acquire a first image and a second image of the ceiling captured by the camera, extract a first geometric feature from the first image and a second geometric feature from the second image; acquire a relative pose between a first pose sensed by the pose sensor when the mobile device captures the first image and a second pose sensed by the pose sensor when the mobile device captures the second image; based on the relative pose and the camera intrinsic parameters of the mobile device, project the first geometric feature and the second geometric feature onto the same coordinate system with the height of the ceiling as the height of the first geometric feature and the second geometric feature, to obtain a feature projection image; determine a first geometric feature that coincides with the second geometric feature on the feature projection image as a target first geometric feature; and determine the target pose of the mobile device when it captures the first image based on the target first geometric feature.

[0149] In one possible implementation, the processor 303, based on the relative pose and the camera intrinsics of the mobile device, projects the first and second geometric features onto the same coordinate system, using the ceiling height as the height of the first and second geometric features, to obtain a feature projection image, including:

[0150] Based on the relative pose and the camera intrinsics of the mobile device, establish a conversion relationship between the first image on the ceiling and the second image on the ceiling;

[0151] Based on the transformation relationship, the second geometric feature is projected onto the first image to obtain a feature projection image;

[0152] The geometric features include line features;

[0153] The extraction of the first geometric features of the first image and the second geometric features of the second image includes:

[0154] Line features with an angle greater than a preset angle threshold to the direction of motion are extracted from the first image and used as first geometric features; and line features with an angle greater than the preset angle threshold to the direction of motion are extracted from the second image and used as second geometric features; wherein, the direction of motion is the direction of movement of the mobile device from capturing the second image to capturing the first image.

[0155] The processor is further configured to extract line features that are perpendicular to and connected to the first geometric feature of the target in the first image, as supplementary first geometric features;

[0156] Determining the target pose of the mobile device when it captures the first image based on the first geometric features of the target includes:

[0157] Based on the target's first geometric features and the supplementary first geometric features, the target pose when the mobile device captures the first image is determined.

[0158] The step of determining the first geometric feature that coincides with the second geometric feature on the feature projection image as the target first geometric feature includes:

[0159] On the feature projection image, a first geometric feature with an overlap degree greater than a preset overlap threshold with the second geometric feature is determined and designated as the target first geometric feature. This application also provides a global positioning device, such as... Figure 4 As shown, the device includes:

[0160] The acquisition module 401 is used to acquire a first image and a second image of the ceiling taken by the mobile device in different poses, and the relative pose between the poses of the mobile device when it takes the first image and the second image.

[0161] Extraction module 402 is used to extract the first geometric features of the first image and the second geometric features of the second image;

[0162] The projection module 403 is used to project the first geometric feature and the second geometric feature onto the same coordinate system based on the relative pose and the camera intrinsic parameters of the mobile device, with the height of the ceiling as the height of the first geometric feature and the second geometric feature, to obtain a feature projection image.

[0163] The target feature determination module 404 is used to determine a first geometric feature that coincides with the second geometric feature on the feature projection image, as the target first geometric feature;

[0164] The target pose determination module 405 is used to determine the target pose when the mobile device captures the first image based on the first geometric features of the target.

[0165] In this embodiment, firstly, the first geometric features of the first image captured from different poses and the second geometric features of the second image are extracted. Then, the first and second geometric features are projected onto the same coordinate system. Since the projection is based on the height of the ceiling, only the second geometric features whose actual height is the ceiling will overlap with the first geometric features during projection. These overlapping target first geometric features are the features on the ceiling, thus quickly filtering out features that do not belong to the ceiling. Finally, the target pose of the mobile device when capturing the first image is determined based on the target geometric features of the ceiling. Since features that do not belong to the ceiling can be quickly filtered out simply by the feature projection deviation of the images captured from different poses, complex calculations are not required, thus improving the efficiency of global positioning.

[0166] In one possible implementation, the projection module 403, based on the relative pose and the camera intrinsics of the mobile device, projects the first and second geometric features onto the same coordinate system, using the ceiling height as the height of the first and second geometric features, to obtain a feature projection image, including:

[0167] Based on the relative pose and the camera intrinsics of the mobile device, establish a conversion relationship between the first image on the ceiling and the second image on the ceiling;

[0168] Based on the transformation relationship, the second geometric feature is projected onto the first image to obtain a feature projection image.

[0169] In one possible implementation, the geometric features include line features;

[0170] The extraction module 402 extracts the first geometric features of the first image and the second geometric features of the second image, including:

[0171] Line features with an angle greater than a preset angle threshold to the direction of motion are extracted from the first image and used as first geometric features; and line features with an angle greater than the preset angle threshold to the direction of motion are extracted from the second image and used as second geometric features; wherein, the direction of motion is the direction of movement of the mobile device from capturing the second image to capturing the first image.

[0172] The device further includes:

[0173] The supplementary feature determination module 406 is used to extract line features that are perpendicular to and connected to the target first geometric feature in the first image, as supplementary first geometric features;

[0174] The target pose determination module 405 determines the target pose when the mobile device captures the first image based on the first geometric features of the target, including:

[0175] Based on the target's first geometric features and the supplementary first geometric features, the target pose when the mobile device captures the first image is determined.

[0176] The target feature determination module 404 determines a first geometric feature that coincides with the second geometric feature on the feature projection image, as the target first geometric feature, including:

[0177] On the feature projection image, a first geometric feature with an overlap degree greater than a preset overlap threshold is determined as the target first geometric feature.

[0178] This application also provides an electronic device, such as... Figure 5 As shown, it includes:

[0179] Memory 501 is used to store computer programs;

[0180] When processor 502 executes the program stored in memory 501, it performs the following steps:

[0181] Acquire a first image and a second image of the ceiling taken by the mobile device in different poses, and the relative pose between the poses of the mobile device when it takes the first image and the second image;

[0182] Extract the first geometric features of the first image and the second geometric features of the second image;

[0183] Based on the relative pose and the camera intrinsics of the mobile device, the first geometric feature and the second geometric feature are projected onto the same coordinate system with the height of the ceiling as the height of the first geometric feature and the second geometric feature, to obtain a feature projection image.

[0184] On the feature projection image, a first geometric feature that coincides with the second geometric feature is determined as the target first geometric feature;

[0185] Based on the first geometric features of the target, the target pose when the mobile device captures the first image is determined.

[0186] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 502, communication interface, and memory 501 communicating with each other via the communication bus.

[0187] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0188] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0189] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0190] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0191] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the global positioning methods described above.

[0192] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the global positioning methods described above.

[0193] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0194] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0195] The various embodiments in this specification are described in a related manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0196] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A global positioning method, characterized in that, The method includes: Acquire a first image and a second image of the ceiling taken by the mobile device in different poses, and the relative pose between the poses of the mobile device when it takes the first image and the second image; Extract the first geometric features of the first image and the second geometric features of the second image; Based on the relative pose and the camera intrinsics of the mobile device, the first geometric feature and the second geometric feature are projected onto the same coordinate system with the height of the ceiling as the height of the first geometric feature and the second geometric feature, to obtain a feature projection image. On the feature projection image, a first geometric feature that coincides with the second geometric feature is determined as the target first geometric feature; Based on the first geometric features of the target, the target pose when the mobile device captures the first image is determined.

2. The method according to claim 1, characterized in that, The step of projecting the first and second geometric features onto the same coordinate system based on the relative pose and the camera intrinsics of the mobile device, with the ceiling height as the height of the first and second geometric features, to obtain a feature projection image includes: Based on the relative pose and the camera intrinsics of the mobile device, establish a conversion relationship between the first image on the ceiling and the second image on the ceiling; Based on the transformation relationship, the second geometric feature is projected onto the first image to obtain a feature projection image.

3. The method according to claim 1, characterized in that, The geometric features include line features; The extraction of the first geometric features of the first image and the second geometric features of the second image includes: Line features with an angle greater than a preset angle threshold to the direction of motion are extracted from the first image and used as first geometric features; and line features with an angle greater than the preset angle threshold to the direction of motion are extracted from the second image and used as second geometric features; wherein, the direction of motion is the direction of movement of the mobile device from capturing the second image to capturing the first image.

4. The method according to claim 3, characterized in that, The method further includes: Extract line features that are perpendicular to and connected to the first geometric feature of the target from the first image, as supplementary first geometric features; Determining the target pose of the mobile device when it captures the first image based on the first geometric features of the target includes: Based on the target's first geometric features and the supplementary first geometric features, the target pose when the mobile device captures the first image is determined.

5. The method according to claim 1, characterized in that, The step of determining the first geometric feature that coincides with the second geometric feature on the feature projection image as the target first geometric feature includes: On the feature projection image, a first geometric feature with an overlap degree greater than a preset overlap threshold with the second geometric feature is determined and used as the target first geometric feature.

6. A mobile device, characterized in that, The mobile device includes: A pose sensor is used to monitor the pose of the mobile device in real time. A camera for photographing the ceiling when the mobile device is in different poses; The processor is configured to acquire a first image and a second image of the ceiling captured by the camera; extract a first geometric feature from the first image and a second geometric feature from the second image; acquire a relative pose between a first pose sensed by the pose sensor when the mobile device captures the first image and a second pose sensed by the pose sensor when the mobile device captures the second image; based on the relative pose and the camera intrinsic parameters of the mobile device, project the first geometric feature and the second geometric feature onto the same coordinate system with the height of the ceiling as the height of the first geometric feature and the second geometric feature to obtain a feature projection image; determine a first geometric feature that coincides with the second geometric feature on the feature projection image as a target first geometric feature; and determine the target pose of the mobile device when it captures the first image based on the target first geometric feature.

7. The mobile device according to claim 6, characterized in that, The processor, based on the relative pose and the camera intrinsics of the mobile device, projects the first and second geometric features onto the same coordinate system, using the ceiling height as the height of the first and second geometric features, to obtain a feature projection image, including: Based on the relative pose and the camera intrinsics of the mobile device, establish a conversion relationship between the first image on the ceiling and the second image on the ceiling; Based on the transformation relationship, the second geometric feature is projected onto the first image to obtain a feature projection image; The geometric features include line features; The extraction of the first geometric features of the first image and the second geometric features of the second image includes: Line features with an angle greater than a preset angle threshold to the direction of motion are extracted from the first image and used as first geometric features; and line features with an angle greater than the preset angle threshold to the direction of motion are extracted from the second image and used as second geometric features; wherein, the direction of motion is the direction of movement of the mobile device from capturing the second image to capturing the first image. The processor is further configured to extract line features that are perpendicular to and connected to the first geometric feature of the target in the first image, as supplementary first geometric features; Determining the target pose of the mobile device when it captures the first image based on the first geometric features of the target includes: Based on the target's first geometric features and the supplementary first geometric features, the target pose when the mobile device captures the first image is determined. The step of determining the first geometric feature that coincides with the second geometric feature on the feature projection image as the target first geometric feature includes: On the feature projection image, a first geometric feature with an overlap degree greater than a preset overlap threshold with the second geometric feature is determined and used as the target first geometric feature.

8. A global positioning device, characterized in that, The device includes: The acquisition module is used to acquire a first image and a second image of the ceiling taken by the mobile device in different poses, as well as the relative pose between the poses of the mobile device when it takes the first image and the second image. The extraction module is used to extract the first geometric features of the first image and the second geometric features of the second image; The projection module is used to project the first geometric feature and the second geometric feature onto the same coordinate system based on the relative pose and the camera intrinsic parameters of the mobile device, with the height of the ceiling as the height of the first geometric feature and the second geometric feature, to obtain a feature projection image. The target feature determination module is used to determine a first geometric feature that coincides with the second geometric feature on the feature projection image, as the target first geometric feature; The target pose determination module is used to determine the target pose when the mobile device captures the first image based on the first geometric features of the target.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-5.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-5.

Images