A method, system, apparatus, and media for three-dimensional map gravity alignment

Abstract

The application relates to a method, system, device and medium for three-dimensional map gravity alignment, which comprises the following steps: judging whether there is a prior pose of a gravity coordinate system at present, if not, obtaining a first gravity axis vector for three-dimensional map gravity alignment according to an average gravity axis estimation algorithm of visual images in multiple shooting dimensions, if yes, obtaining a second gravity axis vector for three-dimensional map gravity alignment according to the pose of the image under the map reconstruction coordinate system and the pose of the image under the prior gravity coordinate system while obtaining the first gravity axis vector, calculating the included angle between the first gravity axis vector and the second gravity axis vector, and determining the final gravity axis vector by comparing the size between the included angle and a preset threshold value; and calculating a rotation matrix required by three-dimensional map gravity alignment according to the gravity axis vector obtained under different conditions, and completing the final gravity alignment. Through the application, the accuracy of gravity alignment is improved, and the use scene is expanded.

Description

Technical Field

[0001] This application relates to the field of 3D reconstruction technology, and in particular to a method, system, apparatus and medium for gravity alignment of 3D maps. Background Technology

[0002] In 3D reconstruction technology, gravity alignment of 3D maps is essential. For example, in AR navigation, if the given map is gravity aligned, the height of navigation guide elements can be easily set; when creating certain AR content, if the map is gravity aligned, it is easy to design content emerging effects from bottom to top.

[0003] In response to this, commonly used gravity alignment methods in related technologies include: The first method involves aligning the reconstructed map using gravity. However, this purely visual gravity alignment method has several limitations. Many real-world scenarios do not conform to the Manhattan world assumption; for example, there may be no obvious straight lines parallel to gravity, or the camera may not be held upright but vertically or even upside down. Both of these situations can lead to errors or even failures in gravity alignment. The second method utilizes devices with gravity detection capabilities, such as mobile phones with IMUs (Inertial Measurement Units). During shooting, a gravity-aligned coordinate system is constructed based on the IMU's gravity detection information, and the pose of each frame in this coordinate system is recorded. Finally, the reconstructed map coordinate system is aligned to the gravity-aligned coordinate system. The disadvantage of this approach is that it requires a specific device with an IMU and a specific algorithm to estimate the shooting pose in real time during shooting. However, the real-time estimated shooting pose may be unstable or contain significant errors, which can introduce substantial errors into the subsequent gravity alignment of the map coordinate system.

[0004] Therefore, no effective solution has yet been proposed to address the issues of low accuracy in gravity alignment of 3D maps and equipment limitations in the aforementioned related technologies. Summary of the Invention

[0005] This application provides a method, system, apparatus, and medium for gravity alignment of three-dimensional maps, in order to at least solve the problems of low accuracy and equipment limitations in gravity alignment of three-dimensional maps in related technologies.

[0006] In a first aspect, embodiments of this application provide a method for gravity alignment of a three-dimensional map, the method comprising:

[0007] Determine whether there is a prior pose in the gravity coordinate system. If not, obtain the first gravity axis vector for gravity alignment of the 3D map based on the average gravity axis estimation algorithm of the visual images under multiple shooting dimensions.

[0008] If present, while obtaining the first gravity axis vector, calculate and obtain the second gravity axis vector for gravity alignment of the three-dimensional map based on the pose of the image in the map reconstruction coordinate system and the pose of the image in the prior gravity coordinate system. Calculate the angle between the first gravity axis vector and the second gravity axis vector, and determine the final gravity axis vector by comparing the angle with a preset threshold.

[0009] Based on the gravity axis vectors obtained under different conditions, calculate the rotation matrix required for gravity alignment of the 3D map, and complete the final gravity alignment.

[0010] In some embodiments, obtaining a first gravity axis vector for gravity alignment of a 3D map based on an average gravity axis estimation algorithm of visual images from multiple shooting dimensions includes:

[0011] Acquire multiple frames of images, along with the corresponding camera intrinsics, image pose, and image rotation angle;

[0012] The images are divided into different groups based on the image rotation angle, and the gravity axis vector of each group is calculated using a custom algorithm.

[0013] By using a custom mean estimation method to fuse all gravity axis vectors, the optimal gravity axis vector estimate is obtained and used as the first gravity axis vector.

[0014] In some embodiments, the image rotation angle includes:

[0015] Within a certain threshold range, the vertical directed line segments of each image are detected. When the direction of the directed line segment is from bottom to top, the image is a positive horizontal imaging angle. When the direction of the directed line segment is from top to bottom, the image is an inverse horizontal imaging angle.

[0016] Within a certain threshold range, the horizontal directed line segments of each image are detected. When the direction of the directed line segment is from right to left, the image is a positive vertical imaging angle. When the direction of the directed line segment is from left to right, the image is a negative vertical imaging angle.

[0017] In this image, the orientation of the directed line segments is referenced to the direction of Earth's gravity, with the positive direction of Earth's gravity being from bottom to top.

[0018] In some embodiments, calculating the gravity axis vector for each group using a custom algorithm includes:

[0019] The hidden point is calculated based on the directed line segment. Based on the hidden point, camera intrinsic parameters, and camera extrinsic parameters, the three-dimensional vector value of the average orientation of the line containing the directed line segment is calculated. The three-dimensional vector is then normalized to obtain the gravity axis vector of the image.

[0020] In some embodiments, the fusion calculation of all gravity axis vectors using a custom mean estimation method includes:

[0021] Select one vector from all gravity axis vectors as a reference vector, calculate the angle between the remaining vectors and the reference vector, and filter out gravity axis vectors with angles less than a preset angle threshold to form a set of gravity axis vectors;

[0022] The number of elements in the set is counted, and it is determined whether the number is greater than the number of elements in the preset vector set. If it is greater, the preset vector set is set equal to the gravity axis vector set.

[0023] Determine whether the number of elements in the preset vector set is greater than a preset threshold. If the threshold is met, proceed directly to the next step. If the threshold is not met, determine whether the number of repeated iterations has reached the preset maximum number of iterations. If the threshold is not met, continue to repeat the iterations. If the threshold is met, proceed directly to the next step.

[0024] The first gravity axis vector is obtained by calculating the mean of the vectors in the preset vector set, and the calculation formula is as follows:

[0025]

[0026] Where, N inlier This refers to the number of elements in the set of vectors composed of gravity axis vectors whose included angle is less than a preset included angle threshold. It refers to the gravity axis vector of the i-th image.

[0027] In some embodiments, calculating a second gravity axis vector for 3D map gravity alignment based on the image's pose in the map reconstruction coordinate system and the image's pose in the prior gravity coordinate system includes:

[0028] The poses of each frame image in the map reconstruction coordinate system and the poses of the image in the prior gravity coordinate system are obtained. The poses in the map reconstruction coordinate system are aligned with gravity according to the pitch and roll angles in the prior poses of the gravity coordinate system to obtain the gravity axis vectors of each frame image.

[0029] The gravity axis vectors of all images are fused and calculated using a custom mean estimation method to obtain the second gravity axis vector.

[0030] In some embodiments, determining the final gravity axis vector by comparing the included angle with a preset threshold includes:

[0031] When the included angle is greater than a preset threshold, the second gravity axis vector is determined to be the final gravity axis vector;

[0032] When the included angle is less than a preset threshold, the first gravity axis vector and the second gravity axis vector are averaged and the result is the final gravity axis vector.

[0033] Secondly, embodiments of this application provide a system for gravity alignment of three-dimensional maps, the system comprising:

[0034] The judgment and calculation module is used to determine whether there is a prior pose in the gravity coordinate system. If not, it obtains the first gravity axis vector for gravity alignment of the 3D map based on the average gravity axis estimation algorithm of the visual images under multiple shooting dimensions.

[0035] If present, while obtaining the first gravity axis vector, calculate and obtain the second gravity axis vector for gravity alignment of the three-dimensional map based on the pose of the image in the map reconstruction coordinate system and the pose of the image in the prior gravity coordinate system. Calculate the angle between the first gravity axis vector and the second gravity axis vector, and determine the final gravity axis vector by comparing the angle with a preset threshold.

[0036] The gravity alignment module is used to calculate the rotation matrix required for gravity alignment of the 3D map based on the gravity axis vectors obtained under different conditions, and to complete the final gravity alignment.

[0037] Thirdly, embodiments of this application provide an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0038] Fourthly, embodiments of this application provide a storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.

[0039] Compared to related technologies, the 3D map gravity alignment method provided in this application determines whether there is a prior gravity coordinate system pose. If not, it obtains a first gravity axis vector for 3D map gravity alignment based on the average gravity axis estimation algorithm of visual images under multiple shooting dimensions. If there is a gravity axis vector, it calculates a second gravity axis vector for 3D map gravity alignment based on the pose of the image in the map reconstruction coordinate system and the pose of the image in the prior gravity coordinate system while obtaining the first gravity axis vector. It calculates the angle between the first gravity axis vector and the second gravity axis vector and determines the final gravity axis vector by comparing the angle with a preset threshold. Based on the gravity axis vectors obtained under different conditions, it calculates the rotation matrix required for 3D map gravity alignment and completes the final gravity alignment.

[0040] This application combines two gravity alignment schemes, achieving complementary advantages and mutual verification. The pure visual gravity alignment scheme is always executed. If a priori gravity coordinate system pose exists, the gravity alignment based on the pose prior and the pure visual gravity alignment scheme are executed simultaneously. The pure visual gravity alignment scheme automatically corrects the gravity alignment result based on the prior rotation of the camera image during shooting, while the gravity alignment scheme based on the pose prior utilizes the accuracy of the pitch and roll angles in the prior pose to calculate the gravity axis vector. This solves the problems of low accuracy and equipment limitations in 3D map gravity alignment in related technologies, improves the accuracy of gravity alignment, and expands its application scenarios. Attached Figure Description

[0041] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0042] Figure 1 This is a flowchart of a method for gravity alignment of a three-dimensional map according to an embodiment of this application;

[0043] Figure 2 These are schematic diagrams illustrating imaging angles of different visual images according to embodiments of this application;

[0044] Figure 3 This is a structural block diagram of a three-dimensional map gravity alignment system according to an embodiment of this application;

[0045] Figure 4 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application. Furthermore, it is understood that although the efforts made in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, modifications to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0047] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0048] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application means two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The terms “first,” “second,” “third,” etc., used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0049] A 3D map refers to a map that represents a scene in three-dimensional coordinates, such as a 3D point cloud map or a 3D mesh map. In this application's embodiments, the 3D map specifically refers to a 3D map constructed through visual imaging. Its characteristic is that, in addition to the final point cloud or mesh map, it can also acquire the 6DoF pose of the camera body in the map coordinate system at the time each image is captured.

[0050] Gravity alignment of a 3D map refers to aligning a coordinate axis in the map coordinate system with the direction of gravity or the opposite direction of gravity. In this application, the gravity alignment method is defined as aligning the Z-axis in the map coordinate system with the opposite direction of gravity, i.e., the Z-axis is perpendicular to the horizontal plane of the earth and pointing upwards.

[0051] This embodiment provides a method for gravity alignment of three-dimensional maps. Figure 1 This is a flowchart of a method for three-dimensional map gravity alignment according to an embodiment of this application, such as... Figure 1As shown, the process includes the following steps:

[0052] Step S101: Determine whether there is a prior pose of the gravity coordinate system. If not, obtain the first gravity axis vector for gravity alignment of the 3D map based on the average gravity axis estimation algorithm of the visual images under multiple shooting dimensions.

[0053] In this embodiment, it is first determined whether there is a prior pose of the gravity coordinate system. If not, the first gravity axis vector for gravity alignment of the 3D map is obtained directly based on the average gravity axis estimation algorithm of the visual images under multiple shooting dimensions.

[0054] Preferably, in this embodiment, obtaining the first gravity axis vector for gravity alignment of the 3D map based on the average gravity axis estimation algorithm of visual images under multiple shooting dimensions includes:

[0055] S1. Acquire multiple frames of images, as well as the camera intrinsics, image pose, and image rotation angle of the corresponding frames;

[0056] S2. Divide the image into different groups according to the image rotation angle, and calculate the gravity axis vector of each group using a custom algorithm;

[0057] S3. All gravity axis vectors are fused and calculated using a custom mean estimation method to obtain the optimal gravity axis vector estimate as the first gravity axis vector.

[0058] It is worth noting that the multiple frames of images acquired in step S1 will produce different imaging rotation angles depending on the actual shooting angle. Figure 2 These are schematic diagrams illustrating imaging angles of different visual images according to embodiments of this application, such as... Figure 2 As shown, the specific process for determining the image imaging rotation angle is as follows:

[0059] 1. If the camera is shooting at a perfectly horizontal angle when taking an image, that is, the sky is above the ground and the ground is below, then during actual imaging, the vertical directional line segments of each image are detected within a certain threshold range. When the direction of the directional line segments is from bottom to top, the image is at a perfectly horizontal imaging angle.

[0060] 2. If the camera is shooting in reverse horizontal direction when taking the image, then during actual imaging, the vertical directional line segments of each image are detected within a certain threshold range. When the direction of the directional line segments is from top to bottom, the image is at the reverse horizontal imaging angle.

[0061] 3. If the camera is shooting perpendicularly when capturing an image, then during actual imaging, the horizontal directed line segments of each image are detected within a certain threshold range. When the direction of the directed line segments is from right to left, the image is at a perpendicular imaging angle.

[0062] 4. If the camera is shooting in reverse perpendicular direction when taking the image, then during actual imaging, the horizontal directed line segments of each image are detected within a certain threshold range. When the direction of the directed line segments is from left to right, the image is at the reverse perpendicular imaging angle.

[0063] In this embodiment, the orientation of the directed line segments in the image is referenced to the direction of Earth's gravity. The positive direction of gravity is set to be from bottom to top. For example, by referring to the positive direction of Earth's gravity, the vertical directed line segments in the image are detected. If their orientation is the positive direction of Earth's gravity, i.e. from bottom to top, then the image has a positive horizontal imaging angle.

[0064] It should be noted that regardless of the shooting angle, the actual image angle after imaging is always within the four rotation angles mentioned above. For example, if the photographer shoots by shifting the image 15 degrees upwards from the horizontal, the actual imaging angle of the resulting image will still be within the range of these four rotation angles. Figure 2 Image quality when photographed at a neutral horizontal angle.

[0065] Specifically, the process of estimating the gravity axis vector for images taken from different shooting angles is as follows: Images from different shooting angles are acquired. If the images are taken from a horizontal or inverted horizontal position, vertical directed line segments from bottom to top are detected in each image; if the images are taken from a vertical or inverted vertical position, horizontal directed line segments from right to left are detected in each image. Then, the gravity axis vector for each group of images is calculated according to step S2. The specific steps include: calculating hidden line points based on the directed line segments in the images from the different shooting groups; and calculating the three-dimensional vector of the average orientation of the lines containing these directed line segments based on the hidden line points, camera intrinsic parameters, and camera extrinsic parameters. This three-dimensional vector is then normalized to obtain the gravity axis vector for each image. According to the Earth's gravity reference direction, when the image is at a positive horizontal imaging angle, the gravity axis vector is g. I =v; when the image is at the inverse horizontal imaging angle, the gravity axis vector is g. I =v; when the image is at a positive perpendicular imaging angle, the gravity axis vector is g. I =v; when the image is at the inverse perpendicular imaging angle, the gravity axis vector is g. I =v, where v is a normalized three-dimensional vector obtained by normalizing the average orientation vector of the straight line containing the directed line segment in different groups of images.

[0066] At this point, the estimated gravity axis vector values ​​for all images can be obtained.

[0067] Then, based on the gravity axis vectors of all images, the optimal gravity axis vector estimate is calculated in step S3. The specific method is as follows:

[0068] (1) Select one vector from all gravity axis vectors as the reference vector;

[0069] (2) Calculate the angle between the remaining vector and the above reference vector. and the included angle Less than the preset included angle threshold The gravity axis vectors are selected to form a set G of gravity axis vectors. cur ;

[0070] (3) Statistical set G cur The number of elements N cur And determine the number N. cur Is it greater than the preset vector set G? best If the number of elements in G is greater than the number of elements in G, then let G be greater than the number of elements in G. best =G cur Conversely, proceed directly to the next step, where G... best The initial value is an empty set;

[0071] (4) Determine the preset vector set G best Is the number of elements greater than the preset threshold N? max If the conditions are met, proceed directly to the next step; otherwise, determine whether the number of iterations from (1) to (4) has reached the preset maximum number of iterations L. max If the condition is not met, continue to iterate and execute (1)-(4). If the condition is met, proceed directly to the next step.

[0072] (5) For the preset vector set G best The first gravity axis vector is obtained by averaging all vectors in the vector. The calculation formula is as follows (1):

[0073]

[0074] Where, N inlier This refers to the number of elements in the set of vectors composed of gravity axis vectors whose included angle is less than a preset included angle threshold. It refers to the gravity axis vector of the i-th image.

[0075] At this point, since there is no prior pose in the gravity coordinate system, the gravity axis vector can only be calculated based on the pure visual image. Therefore, in this embodiment, the first gravity axis vector is calculated. As the final gravity axis vector g F ,Right now

[0076] It should be noted that if a mobile device, such as a mobile phone, is used for shooting, the rotation angle of the image will be written into the video's metadata. Therefore, by simply reading these angles from the video file, the shooting angle of the video can be determined, thereby identifying and classifying the actual imaging angle of the image.

[0077] In this embodiment, the gravity axis vector of each image is calculated by classifying the images according to different imaging angles. Then, a custom mean is calculated based on the gravity axis vectors of all images to obtain the optimal gravity axis vector. That is, this embodiment can automatically correct the gravity alignment result of pure vision based on the rotation prior of the camera screen during shooting, effectively reducing the deviation in calculation and improving robustness.

[0078] Step S102: If present, while obtaining the first gravity axis vector, calculate and obtain the second gravity axis vector for gravity alignment of the three-dimensional map based on the pose of the image in the map reconstruction coordinate system and the pose of the image in the prior gravity coordinate system. Calculate the angle between the first gravity axis vector and the second gravity axis vector, and determine the final gravity axis vector by comparing the angle with a preset threshold.

[0079] Preferably, in this embodiment, if it is determined that there is a prior pose in the gravity coordinate system, then while calculating and obtaining the first gravity axis vector in step S101, a second gravity axis vector for 3D map gravity alignment is calculated based on the image pose in the map reconstruction coordinate system and the image pose in the prior gravity coordinate system. Specifically, this method includes: obtaining the pose of each frame image in the map reconstruction coordinate system and the image pose in the prior gravity coordinate system; then, using the accuracy of the pitch and roll angles in the prior gravity coordinate system pose, performing gravity alignment on the pose in the map reconstruction coordinate system to obtain the gravity axis vectors of each frame image. Since the IMU exists when calculating the prior gravity coordinate system pose, the pitch and roll angles of the camera pose in both the prior gravity coordinate system and the initial map reconstruction coordinate system are very accurate. Therefore, in both the prior gravity coordinate system and the initial map reconstruction coordinate system, the angles between the X, Y, and Z axes of the camera coordinate system and the gravity direction remain unchanged. Furthermore, in the prior gravity coordinate system, the gravity direction is z... a = [0, 0, 1] T Let the pose of each frame of the image in the map reconstruction coordinate system be... Let the pose of each frame in the prior gravity coordinate system be... Based on the invariance principle, the gravity axis vectors of each frame image are obtained by calculating using the following formulas (2)-(5) to align the poses in the map reconstruction coordinate system with gravity.

[0080]

[0081]

[0082]

[0083]

[0084] in, These are the unit direction vectors of the X, Y, and Z axes of the camera coordinate system in the prior gravity coordinate system, respectively. These represent the angles between the X, Y, and Z axes of the camera coordinate system and the direction of gravity, respectively. These are the unit direction vectors of the X, Y, and Z axes of the camera coordinate system in the map reconstruction coordinate system.

[0085] After obtaining the gravity axis vectors of each frame, the custom mean estimation method in step S3 above is used to fuse the gravity axis vectors of all images in this implementation to obtain the second gravity axis vector.

[0086] In this embodiment, a gravity alignment method based on pose prior is used: based on the accuracy of pitch and roll angles in the prior pose, the invariance of the angles between the X, Y, and Z axes of the camera coordinate system and the gravity direction is determined under the gravity prior coordinate system and the initial coordinate system for map reconstruction. This invariance is used to estimate the direction vector of gravity, thereby aligning it to the map coordinate system and improving the accuracy of the estimation.

[0087] At this point, due to the prior pose in the gravity coordinate system, the first gravity axis vector is calculated after the second gravity axis vector is obtained. With the second gravity axis vector The included angle γ g If the angle is greater than a preset threshold, it likely indicates that the current scene does not satisfy the Manhattan world hypothesis, and the first gravity axis vector is obtained based on purely visual images. There is a significant error; therefore, only the second gravity axis vector calculated based on the pose prior method is selected. As the final gravity axis vector g F , that is to say If the angle is less than or equal to the preset threshold, it means that the first gravity axis vector obtained by the two methods is... With the second gravity axis vector Both are relatively good, therefore, the average of the two is taken as the final gravity axis vector g. F ,Right now

[0088] Step S103: Based on the gravity axis vectors obtained under different conditions, calculate the rotation matrix required for gravity alignment of the 3D map, and complete the final gravity alignment.

[0089] Preferably, the final gravity axis vector g under different conditions is calculated through the above steps S101 and S102. F Then, the rotation matrix required for gravity alignment of the 3D map is calculated to complete the final gravity alignment. The calculation formula is shown in equation (6) below:

[0090] ΔR=f a (g F ,z) (6)

[0091] Where z = [0, 0, 1] T f a It is a general function used to calculate the rotation matrix required to align two vectors, without specific limitations.

[0092] Through steps S101 to S103, this embodiment combines two gravity alignment schemes, achieving complementary advantages and mutual verification. Specifically, the pure visual gravity alignment scheme is always executed. If a priori gravity coordinate system pose exists, the gravity alignment based on the pose prior and the pure visual gravity alignment scheme are executed simultaneously. The pure visual gravity alignment scheme automatically corrects the pure visual gravity alignment result based on the priori rotation of the camera image during shooting, while the gravity alignment scheme based on the pose prior utilizes the accuracy of the pitch and roll angles in the priori pose to calculate the gravity axis vector. This solves the problems of low accuracy and equipment limitations in 3D map gravity alignment in related technologies, improves the accuracy of gravity alignment, and expands the application scenarios.

[0093] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0094] This embodiment also provides a system for three-dimensional map gravity alignment, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0095] Figure 3 This is a structural block diagram of a three-dimensional map gravity alignment system according to an embodiment of this application, such as... Figure 3 As shown, the system includes a judgment calculation module 31 and a gravity alignment module 32:

[0096] The judgment and calculation module 31 is used to determine whether there is a prior pose in the gravity coordinate system. If not, it obtains the first gravity axis vector for gravity alignment of the 3D map based on the average gravity axis estimation algorithm of the visual images under multiple shooting dimensions. If there is, it calculates the second gravity axis vector for gravity alignment of the 3D map based on the pose of the image in the map reconstruction coordinate system and the pose of the image in the prior gravity coordinate system while obtaining the first gravity axis vector. It calculates the angle between the first gravity axis vector and the second gravity axis vector and determines the final gravity axis vector by comparing the angle with a preset threshold. The gravity alignment module 32 is used to calculate the rotation matrix required for gravity alignment of the 3D map based on the gravity axis vectors obtained under different conditions and completes the final gravity alignment.

[0097] Through the aforementioned system, this embodiment combines two gravity alignment schemes, achieving complementary advantages and mutual verification. Specifically, the pure visual gravity alignment scheme is always executed. If a priori gravity coordinate system pose exists, the gravity alignment based on the pose prior and the pure visual gravity alignment scheme are executed simultaneously. The pure visual gravity alignment scheme automatically corrects the pure visual gravity alignment result based on the priori rotation of the camera image during shooting, while the gravity alignment scheme based on the pose prior utilizes the accuracy of the pitch and roll angles in the priori pose to calculate the gravity axis vector. This solves the problems of low accuracy and equipment limitations in 3D map gravity alignment in related technologies, improves the accuracy of gravity alignment, and expands the application scenarios.

[0098] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0099] Furthermore, it should be noted that the aforementioned modules can be either functional modules or program modules, and can be implemented through software or hardware. For modules implemented in hardware, these modules can reside in the same processor; alternatively, they can be located in different processors in any combination.

[0100] This embodiment also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0101] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0102] Furthermore, in conjunction with the three-dimensional map gravity alignment method in the above embodiments, this application embodiment can provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the three-dimensional map gravity alignment methods in the above embodiments.

[0103] In one embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, memory, a network interface, a display screen, and an input device connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, it implements a method for gravity alignment of a three-dimensional map. The display screen may be a liquid crystal display (LCD) or an e-ink display. The input device may be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0104] In one embodiment, Figure 4 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application, such as... Figure 4 As shown, an electronic device is provided, which can be a server, and its internal structure diagram can be as follows. Figure 4 As shown, the electronic device includes a processor, a network interface, internal memory, and non-volatile memory connected via an internal bus. The non-volatile memory stores an operating system, computer programs, and a database. The processor provides computing and control capabilities, the network interface communicates with external terminals via a network connection, the internal memory provides an environment for the operation of the operating system and computer programs, the computer programs are executed by the processor to implement a method for three-dimensional map gravity alignment, and the database stores data.

[0105] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0106] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0107] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0108] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for gravity alignment of a three-dimensional map, characterized in that, The method includes: Determine whether there is a prior pose in the gravity coordinate system. If not, obtain the first gravity axis vector for gravity alignment of the 3D map based on the average gravity axis estimation algorithm of the visual images under multiple shooting dimensions. If present, while obtaining the first gravity axis vector, calculate and obtain the second gravity axis vector for gravity alignment of the three-dimensional map based on the pose of the image in the map reconstruction coordinate system and the pose of the image in the prior gravity coordinate system. Calculate the angle between the first gravity axis vector and the second gravity axis vector, and determine the final gravity axis vector by comparing the angle with a preset threshold. Based on the gravity axis vectors obtained under different conditions, calculate the rotation matrix required for gravity alignment of the 3D map, and complete the final gravity alignment.

2. The method according to claim 1, characterized in that, Based on the average gravity axis estimation algorithm of visual images from multiple shooting dimensions, the first gravity axis vector for gravity alignment of the 3D map is obtained, including: Acquire multiple frames of images, along with the corresponding camera intrinsics, image pose, and image rotation angle; The images are divided into different groups based on the image rotation angle, and the gravity axis vector of each group is calculated using a custom algorithm. By using a custom mean estimation method to fuse all gravity axis vectors, the optimal gravity axis vector estimate is obtained and used as the first gravity axis vector.

3. The method according to claim 2, characterized in that, Image rotation angles include: Within a certain threshold range, the vertical directed line segments of each image are detected. When the direction of the directed line segment is from bottom to top, the image is a positive horizontal imaging angle. When the direction of the directed line segment is from top to bottom, the image is an inverse horizontal imaging angle. Within a certain threshold range, the horizontal directed line segments of each image are detected. When the direction of the directed line segment is from right to left, the image is a positive vertical imaging angle. When the direction of the directed line segment is from left to right, the image is a negative vertical imaging angle. In this image, the orientation of the directed line segments is referenced to the direction of Earth's gravity, with the positive direction of Earth's gravity being from bottom to top.

4. The method according to claim 2 or 3, characterized in that, The gravity axis vector for each group is calculated using a custom algorithm, including: The hidden point is calculated based on the directed line segment. Based on the hidden point, camera intrinsic parameters, and camera extrinsic parameters, the three-dimensional vector value of the average orientation of the line containing the directed line segment is calculated. The three-dimensional vector is then normalized to obtain the gravity axis vector of the image.

5. The method according to claim 1, characterized in that, The calculation of all gravity axis vectors is performed using a custom mean estimation method, including: Select one vector from all gravity axis vectors as a reference vector, calculate the angle between the remaining vectors and the reference vector, and filter out gravity axis vectors with angles less than a preset angle threshold to form a set of gravity axis vectors; The number of elements in the set is counted, and it is determined whether the number is greater than the number of elements in the preset vector set. If it is greater, the preset vector set is set equal to the gravity axis vector set. Determine whether the number of elements in the preset vector set is greater than a preset threshold. If the threshold is met, proceed directly to the next step. If the threshold is not met, determine whether the number of repeated iterations has reached the preset maximum number of iterations. If the threshold is not met, continue to repeat the iterations. If the threshold is met, proceed directly to the next step. The first gravity axis vector is obtained by calculating the mean of the vectors in the preset vector set, and the calculation formula is as follows: Where, N inlier This refers to the number of elements in the set of vectors composed of gravity axis vectors whose included angle is less than a preset included angle threshold. It refers to the gravity axis vector of the i-th image.

6. The method according to claim 1, characterized in that, The second gravity axis vector for 3D map gravity alignment is calculated based on the image's pose in the map reconstruction coordinate system and the image's pose in the prior gravity coordinate system. The poses of each frame image in the map reconstruction coordinate system and the poses of the image in the prior gravity coordinate system are obtained. The poses in the map reconstruction coordinate system are aligned with gravity according to the pitch and roll angles in the prior poses of the gravity coordinate system to obtain the gravity axis vectors of each frame image. The gravity axis vectors of all images are fused and calculated using a custom mean estimation method to obtain the second gravity axis vector.

7. The method according to claim 1, characterized in that, The final gravity axis vector is determined by comparing the included angle with a preset threshold. When the included angle is greater than a preset threshold, the second gravity axis vector is determined to be the final gravity axis vector; When the included angle is less than a preset threshold, the first gravity axis vector and the second gravity axis vector are averaged and the result is the final gravity axis vector.

8. A system for gravity alignment of a three-dimensional map, characterized in that, The system includes: The judgment and calculation module is used to determine whether there is a prior pose in the gravity coordinate system. If not, it obtains the first gravity axis vector for gravity alignment of the 3D map based on the average gravity axis estimation algorithm of the visual images under multiple shooting dimensions. If present, while obtaining the first gravity axis vector, calculate and obtain the second gravity axis vector for gravity alignment of the three-dimensional map based on the pose of the image in the map reconstruction coordinate system and the pose of the image in the prior gravity coordinate system. Calculate the angle between the first gravity axis vector and the second gravity axis vector, and determine the final gravity axis vector by comparing the angle with a preset threshold. The gravity alignment module is used to calculate the rotation matrix required for gravity alignment of the 3D map based on the gravity axis vectors obtained under different conditions, and to complete the final gravity alignment.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method of any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the method of any one of claims 1 to 7 when it is run.

Images