Visual SLAM method and device based on structural line, computer equipment and storage medium

Abstract

The invention relates to the technical field of image processing, and discloses a visual SLAM method and device based on a structural line, computer equipment and a storage medium, and the method comprises the steps: obtaining a target image and I MU data; performing pre-integration on the I-MU data according to the I-MU motion model to obtain an I-MU residual error; determining each hypothetical vanishing point combination according to line features in the target image; based on a first hypothesis vanishing point, a second hypothesis vanishing point and a third hypothesis vanishing point, classifying the line features to obtain a structural line; and performing sliding window optimization based on the I MU residual error, the structure line and an SLAM algorithm to estimate the pose. According to the method, the hypothesis vanishing point is extracted, the structural line is distinguished by using the hypothesis vanishing point, and the pose is estimated by using the structural line I MU residual SLAM algorithm, so that the positioning precision and the mapping quality can be improved, and the method can be used without any hypothesis such as the Manhattan world or the Atrate world, and is suitable for indoor and outdoor complex scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular, to a vision SLAM method, device, computer device, and storage medium based on structural lines. Background Art

[0002] Simultaneous Localization and Mapping (SLAM) has been widely applied in the field of intelligent robots, such as autonomous driving, rescue, and augmented reality. Since cameras and inertial measurement units (IMUs) are low-cost and efficient sensors, VIO systems can overcome the disadvantages of these two sensors and improve the accuracy and robustness of localization.

[0003] However, point features have some disadvantages. First, point features are not robust enough in low-texture environments such as corridors. In addition, they are sensitive to light changes. To complement the deficiencies of point features, line-based algorithms have been proposed. Line features can also be used in environments with less texture (such as corridors). In addition, since a line consists of multiple points, the features are likely to be retained even when light changes occur.

[0004] Currently, according to the type of line features, line features can be divided into two categories: unstructured lines and structured lines. Unstructured lines have stronger generality and robustness because they can be used in different types of environments. However, unlike structured lines, unstructured lines do not have effective direction constraints. Structured lines exist in man-made environments, which are abstracted as a set of regions sharing three common dominant directions, namely the so-called Manhattan world. They encode the global direction of the local environment, and these constraints can reduce the drift of the yaw angle. However, when the environment is complex, these constraints are not effective for pose estimation. Since these methods only use structural lines with dominant directions, they are only practical in indoor environments where most assumptions hold. Summary of the Invention

[0005] Based on this, in view of the technical problem of the poor pose estimation effect in the prior art, it is necessary to propose a vision SLAM method, device, computer device, and storage medium based on structural lines.

[0006] In a first aspect, a vision SLAM method based on structural lines is provided. The method includes:

[0007] Obtain a target image and IMU data;

[0008] Pre-integrate the IMU data according to the IMU motion model to obtain an IMU residual;

[0009] Determine each set of hypothesized vanishing points according to the line features in the target image, where the set of hypothesized vanishing points includes a first hypothesized vanishing point, a second hypothesized vanishing point, and a third hypothesized vanishing point;

[0010] Classify the line features based on the first hypothesized vanishing point, the second hypothesized vanishing point, and the third hypothesized vanishing point to obtain structural lines;

[0011] Perform sliding window optimization based on the IMU residuals, the structural lines, and the SLAM algorithm to estimate the pose.

[0012] In a second aspect, a visual SLAM device based on structural lines is provided. The device includes:

[0013] An acquisition module configured to acquire a target image and IMU data;

[0014] An IMU module configured to perform pre-integration on the IMU data according to an IMU motion model to obtain IMU residuals;

[0015] A determination module configured to determine various combinations of hypothesized vanishing points based on line features in the target image, where the combinations of hypothesized vanishing points include a first hypothesized vanishing point, a second hypothesized vanishing point, and a third hypothesized vanishing point;

[0016] A classification module configured to classify the line features based on the first hypothesized vanishing point, the second hypothesized vanishing point, and the third hypothesized vanishing point to obtain structural lines;

[0017] A pose estimation module configured to perform sliding window optimization based on the IMU residuals, the structural lines, and the SLAM algorithm to estimate the pose.

[0018] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-described visual SLAM method based on structural lines are implemented.

[0019] In a fourth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-described visual SLAM method based on structural lines are implemented.

[0020] The visual SLAM method based on structure lines proposed by the present invention obtains a target image and IMU data, then pre-integrates the IMU data according to the IMU motion model to obtain IMU residuals, and then determines various combinations of hypothesized vanishing points based on the line features in the target image, where the combination of hypothesized vanishing points includes a first hypothesized vanishing point, a second hypothesized vanishing point, and a third hypothesized vanishing point. Then, based on the first hypothesized vanishing point, the second hypothesized vanishing point, and the third hypothesized vanishing point, the line features are classified to obtain structure lines. Finally, based on the IMU residuals, the structure lines, and the SLAM algorithm, a sliding window optimization is performed to estimate the pose. The present invention extracts hypothesized vanishing points and uses the hypothesized vanishing points to distinguish structure lines, and uses the structure lines, IMU residuals, and SLAM algorithm to estimate the pose, which can improve the positioning accuracy and mapping quality, and can be used without any assumptions such as Manhattan world or Atlanta world, and is applicable to complex indoor and outdoor scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings.

[0022] Among them:

[0023] Figure 1 It is an application environment diagram of the visual SLAM method based on structure lines in an embodiment;

[0024] Figure 2 It is a flowchart of the visual SLAM method based on structure lines in an embodiment;

[0025] Figure 3 It is a structural block diagram of a visual SLAM device based on structure lines in an embodiment;

[0026] Figure 4 It is a structural block diagram of a computer device in an embodiment;

[0027] Figure 5 It is a structural block diagram of a computer device in another embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs; the terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "comprising" and "having" and any variations thereof in the specification and claims of this application and the above drawings are intended to cover non-exclusive inclusion. The terms "first", "second", etc. in the specification and claims of this application or the above drawings are used to distinguish different objects and not to describe a specific order.

[0029] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive of other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0030] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0031] The visual SLAM method based on structure lines provided by the embodiments of the present invention can be applied in, for example Figure 1In the application environment, the client 110 communicates with the server 120 via a network. The server 120 can receive and obtain the target image and IMU data through the client 110. Then, the server 120 pre-integrates the IMU data according to the IMU motion model to obtain the IMU residual. Next, according to the line features in the target image, various hypothetical vanishing point combinations are determined, where the hypothetical vanishing point combination includes a first hypothetical vanishing point, a second hypothetical vanishing point, and a third hypothetical vanishing point. Then, the server 120 classifies the line features based on the first hypothetical vanishing point, the second hypothetical vanishing point, and the third hypothetical vanishing point to obtain the structural lines. Finally, the server 120 performs sliding window optimization based on the IMU residual, the structural lines, and the SLAM algorithm to estimate the pose. The present invention extracts hypothetical vanishing points and uses the hypothetical vanishing points to distinguish the structural lines, and uses the structural lines, the IMU residual, and the SLAM algorithm to estimate the pose, which can improve the positioning accuracy and mapping quality, and can be used without any assumptions such as Manhattan world or Atlantis world, and is applicable to complex indoor and outdoor scenes. Among them, the client 110 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server 120 can be implemented by an independent server or a server cluster composed of multiple servers. The present invention will be described in detail below through specific embodiments.

[0032] Please refer to Figure 2 as shown in Figure 2 FIG. is a schematic flowchart of a visual SLAM method based on structural lines provided by an embodiment of the present invention, including the following steps:

[0033] Step S101: Obtain the target image and IMU data;

[0034] As an example, in the front end, the target image and IMU data are first processed. For the input IMU data, pre-integration is performed according to the IMU motion model. For the image data, the feature points and feature lines in the current target image are detected and tracked with the previous frame. For example, the LSD algorithm is used to detect the straight lines in the image, and then the LBD descriptor is used for tracking.

[0035] It should be noted that the present invention is mainly based on VINS-Mono. VINS-Mono uses an optimization method to tightly couple the IMU observations with the visual observations of the point features. The present invention adds non-structural lines and structural lines and constructs corresponding constraints.

[0036] Step S102: Pre-integrate the IMU data according to the IMU motion model to obtain the IMU residual;

[0037] Step S103: Determine each combination of hypothesized vanishing points according to the line features in the target image, where each combination of hypothesized vanishing points includes a first hypothesized vanishing point, a second hypothesized vanishing point, and a third hypothesized vanishing point.

[0038] In one embodiment, step S103 includes:

[0039] Step S1031: Select two line features from each of the line features as target line features;

[0040] Step S1032: Perform a cross product based on the two target line features to obtain a first hypothesized vanishing point;

[0041] In this embodiment, a cross product can be performed based on the vector representation of the target line features to obtain the first pixel coordinates of the intersection point as the first hypothesized vanishing point.

[0042] Specifically, first, randomly select two line segments l1 and l2 each time, that is, line features, as target line features. According to the line segment equation, obtain their vector representations. l1 corresponds to (a1, b1, c1), and l2 corresponds to (a2, b2, c2). The cross product calculation can obtain the pixel coordinates (x, y, c3) of the intersection point as the first hypothesized vanishing point.

[0043] Step S1033: Determine the second hypothesized vanishing point based on the outer circle of the orthogonal plane of the first hypothesized vanishing point;

[0044] In this embodiment, given the optical center and focal length, convert the pixel coordinates of the first hypothesized vanishing point to the equivalent spherical coordinate system, and determine the longitude and latitude representation of the first hypothesized vanishing point according to the spherical coordinate system;

[0045] Next, based on the outer circle of the orthogonal plane of the first hypothesized vanishing point, determine the longitude and latitude representation of the second hypothesized vanishing point; finally, based on the longitude and latitude representation of the second hypothesized vanishing point, determine the second pixel coordinates of the second hypothesized vanishing point.

[0046] As an example, given the optical center (x0, y0) and focal length f, the pixel coordinates can be converted to the equivalent spherical coordinate system:

[0047]

[0048] Then convert it to the longitude and latitude representation (φ1, λ1):

[0049]

[0050] Secondly, considering the orthogonality constraint, the second vanishing point must be on the outer circle of the orthogonal plane of υ1. Therefore, uniformly sample 360 hypotheses at intervals of 1° on the outer circle, and each value corresponds to the longitude λ2 of the second hypothesized vanishing point. Its latitude value φ2 can be obtained through the formula:

[0051]

[0052] The pixel coordinates of the second hypothesized vanishing point can be obtained from the longitude and latitude of the second hypothesized vanishing point.

[0053] Step S1033: Cross-multiply the first hypothesized vanishing point and the second hypothesized vanishing point to obtain a third hypothesized vanishing point.

[0054] In this embodiment, the third pixel coordinate of the third hypothesized vanishing point is obtained by cross-multiplying the first pixel coordinate of the first hypothesized vanishing point and the second pixel coordinate of the second hypothesized vanishing point. Then, based on the third pixel coordinate, the longitude and latitude of the third hypothesized vanishing point can be generated.

[0055] Specifically, the third vanishing point υ3 can be obtained by cross-multiplying the vanishing points υ1 and υ2. After iterating N times, 360×N groups of hypotheses can be obtained. Next, the most suitable combination needs to be selected from these hypotheses.

[0056] Step S104: Classify the line features based on the first hypothesized vanishing point, the second hypothesized vanishing point, and the third hypothesized vanishing point to obtain structural lines;

[0057] In this embodiment, first, a polar coordinate grid is established, where the polar coordinate grid is a polar coordinate divided into grids according to preset longitude and latitude in the polar coordinate system; for all line segment pairs on the target image, the longitude and latitude of the intersection point of the line segment pair are determined; the grid response value is calculated based on the longitude and latitude, and the corresponding polar coordinate grid is updated based on the grid response value; based on the estimated longitude and latitude of each hypothesized vanishing point in each combination of hypothesized vanishing points and the updated polar coordinate grid, the best combination of vanishing points is selected; the line features are classified based on the best combination of vanishing points to obtain structural lines.

[0058] As an example, first, a polar coordinate grid is established for quickly querying the response value of the vanishing point. The ranges of longitude and latitude are [0, 2π] and divided by 1°, which can be divided into 90×360 grids. Suppose there are n line segments on the target image, then n(n - 1) pairs of combinations can be obtained by pairwise combination. For all line segment pairs l1 and l2 on the target image, find the longitude and latitude (φ i , λ i ) of their intersection point, and then update the grid response value:

[0059]

[0060] where θ is the smaller included angle between l1 and l2, is the boundary symbol-taking.

[0061] Subsequently, these assumptions need to be verified with the aim of finding a group of assumptions with the maximum line segment combination response among all groups of assumptions. First, for each group of assumptions, calculate the longitude and latitude (φ i , λ i ) corresponding to each vanishing point, where i ∈ 1, 2, 3. Then, conduct a search in the polar coordinate grid and set the sum r of the response values of each group of vanishing points as the line segment combination response of this group of assumptions:

[0062]

[0063] Finally, the group of assumptions with the maximum line segment combination response is selected as the best vanishing point estimation group.

[0064] Subsequently, classify the line segments using the selected best vanishing point combination. For each line segment l in the image, calculate the midpoint of the line segment, and then connect the midpoint to the three vanishing points respectively, denoted as l υ , and calculate the included angle θ between l and l υ :

[0065]

[0066] If one of the θ values is less than the threshold, the line segment l is a structural line; otherwise, it is regarded as a non-structural line.

[0067] Step S105: Perform sliding window optimization based on the IMU residuals, the structural lines, and the SLAM algorithm to estimate the pose.

[0068] In visual inertial SLAM, the pose T i ∈ SE(3), υ i is the velocity in the world coordinate system, b g，i , b a，i are the biases of the gyroscope and accelerometer. Then, define the state as:

[0069]

[0070] Define the state as the set of all key frame states, point features, and line features:

[0071]

[0072] Among them, I, J, and K are the numbers of key frames, point features, and line features respectively.

[0073] In the backend, the overall objective of the sliding window optimization is as follows:

[0074]

[0075] where r0, r I , rp , r l and r v respectively represent the measurement residuals of the marginalization, IMU, feature points, structure lines, and the best vanishing point estimation group. In addition, and respectively represent the observations of the IMU, feature points, structure lines, and the best vanishing point estimation group. is the set of all pre-integrated IMU measurements in the sliding window. and are respectively the sets of measurements of the feature points, structure lines, and the best vanishing point estimation group in the observation frame. and respectively represent the covariance matrices measured by the IMU, feature points, structure lines, and the best vanishing point estimation group. ρ p , ρ l and ρ v respectively represent the robust kernel functions of the measurements of the feature points, structure lines, and the best vanishing point estimation group. Due to the unbounded problem of the vanishing point measurement model, ρ p and ρ l are set as the Huber kernel function, and ρ v is set as the arctangent function. J o is the Jacobian matrix of the prior residual.

[0076] Please refer to Figure 3 As shown, in one embodiment, a vision SLAM device based on structure lines is provided. The device includes: an acquisition module 10 for acquiring a target image and IMU data;

[0077] an IMU module 20 for pre-integrating the IMU data according to the IMU motion model to obtain IMU residuals;

[0078] a determination module 30 for determining various combinations of hypothesized vanishing points according to the line features in the target image, where the combination of hypothesized vanishing points includes a first hypothesized vanishing point, a second hypothesized vanishing point, and a third hypothesized vanishing point;

[0079] a classification module 40 for classifying the line features based on the first hypothesized vanishing point, the second hypothesized vanishing point, and the third hypothesized vanishing point to obtain structure lines;

[0080] a pose estimation module 50 for estimating the pose by performing sliding window optimization based on the IMU residuals, the structure lines, and the SLAM algorithm.

[0081] In one embodiment, the determination module 30 is configured to:

[0082] select two line features from among the respective line features as target line features;

[0083] Cross - multiply based on two target line features to obtain the first hypothesized vanishing point;

[0084] Determine the second hypothesized vanishing point based on the outer circle of the orthogonal plane of the first hypothesized vanishing point;

[0085] Cross - multiply based on the first hypothesized vanishing point and the second hypothesized vanishing point to obtain the third hypothesized vanishing point.

[0086] In one embodiment, the classification module 40 is used for:

[0087] Establish a polar coordinate grid, where the polar coordinate grid is a polar coordinate grid divided according to preset longitude and latitude in the polar coordinate system;

[0088] For all line segment pairs on the target image, determine the longitude and latitude of the intersection point of the line segment pairs;

[0089] Calculate the grid response value based on the longitude and latitude, and update the corresponding polar coordinate grid based on the grid response value;

[0090] Based on the longitude and latitude estimated for each hypothesized vanishing point in each combination of hypothesized vanishing points and the updated polar coordinate grid, screen out the best combination of vanishing points;

[0091] Classify the line features based on the best combination of vanishing points to obtain structural lines.

[0092] In one embodiment, the determination module 30 is used for:

[0093] Perform cross - multiplication based on the vector representation of the target line feature to obtain the first pixel coordinate of the intersection point as the first hypothesized vanishing point.

[0094] In one embodiment, the determination module 30 is used for:

[0095] Given the optical center and focal length, convert the pixel coordinates of the first hypothesized vanishing point to the equivalent spherical coordinate system, and determine the longitude and latitude representation of the first hypothesized vanishing point according to the spherical coordinate system;

[0096] Based on the outer circle of the orthogonal plane of the first hypothesized vanishing point, determine the longitude and latitude representation of the second hypothesized vanishing point;

[0097] Based on the longitude and latitude representation of the second hypothesized vanishing point, determine the second pixel coordinate of the second hypothesized vanishing point.

[0098] In one embodiment, the determination module 30 is used for:

[0099] Perform cross - multiplication on the first pixel coordinate of the first hypothesized vanishing point and the second pixel coordinate of the second hypothesized vanishing point to obtain the third pixel coordinate of the third hypothesized vanishing point.

[0100] In one embodiment, a computer device is provided. The computer device can be a server, and its internal structure diagram can be as shown in Figure 4 . The computer device includes a processor, a memory, a network interface, and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of a visual SLAM method based on structural lines.

[0101] In one embodiment, a computer device is provided. The computer device can be a client, and its internal structure diagram can be as shown in Figure 5 . The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the client side of a visual SLAM method based on structural lines.

[0102] In one embodiment, a computer device is proposed, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented:

[0103] Obtain a target image and IMU data;

[0104] According to the IMU motion model, pre-integrate the IMU data to obtain an IMU residual;

[0105] According to the line features in the target image, determine each combination of hypothesized vanishing points. Among them, the combination of hypothesized vanishing points includes a first hypothesized vanishing point, a second hypothesized vanishing point, and a third hypothesized vanishing point. Then, based on the first hypothesized vanishing point, the second hypothesized vanishing point, and the third hypothesized vanishing point, classify the line features to obtain structural lines;

[0106] Based on the IMU residual, the structural lines, and the SLAM algorithm, perform sliding window optimization to estimate the pose.

[0107] The present invention extracts hypothesized vanishing points and uses the hypothesized vanishing points to distinguish structural lines, and uses the structural line IMU residual SLAM algorithm to estimate the pose, which can improve the positioning accuracy and mapping quality, and can be used without any assumptions such as Manhattan world or Atlantis world, and is applicable to complex indoor and outdoor scenarios.

[0108] In one embodiment, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0109] Obtain a target image and IMU data;

[0110] According to the IMU motion model, pre-integrate the IMU data to obtain IMU residuals;

[0111] According to the line features in the target image, determine each combination of hypothesized vanishing points, where the combination of hypothesized vanishing points includes a first hypothesized vanishing point, a second hypothesized vanishing point, and a third hypothesized vanishing point, and then classify the line features based on the first hypothesized vanishing point, the second hypothesized vanishing point, and the third hypothesized vanishing point to obtain structural lines;

[0112] Based on the IMU residuals, the structural lines, and the SLAM algorithm, perform sliding window optimization to estimate the pose.

[0113] The present invention extracts hypothesized vanishing points and uses the hypothesized vanishing points to distinguish structural lines, and uses the structural line IMU residual SLAM algorithm to estimate the pose, which can improve the positioning accuracy and mapping quality, and can be used without any assumptions such as Manhattan world or Atlantis world, and is applicable to complex indoor and outdoor scenarios.

[0114] It should be noted that for the functions or steps that the above computer-readable storage medium or computer device can achieve, reference can be made to the relevant descriptions on the server side and the client side in the foregoing method embodiments. To avoid repetition, they will not be described in detail here.

[0115] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0116] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0117] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A visual SLAM method based on structure lines, characterized in that The visual SLAM method based on structure lines includes: Obtain a target image and IMU data; According to the IMU motion model, pre-integrate the IMU data to obtain an IMU residual; According to the line features in the target image, determine each combination of hypothesized vanishing points, where the combination of hypothesized vanishing points includes a first hypothesized vanishing point, a second hypothesized vanishing point, and a third hypothesized vanishing point; Classify the line features based on the first hypothesized vanishing point, the second hypothesized vanishing point, and the third hypothesized vanishing point to obtain structure lines; Perform sliding window optimization based on the IMU residual, the structure lines, and the SLAM algorithm to estimate the pose.

2. The visual SLAM method based on structure lines according to claim 1, wherein, The step of determining each combination of hypothesized vanishing points according to the line features in the target image includes: Select two line features from each of the line features as target line features; Perform a cross product based on the two target line features to obtain a first hypothesized vanishing point; Determine the second hypothesized vanishing point based on the outer circle of the orthogonal plane of the first hypothesized vanishing point; Perform a cross product based on the first hypothesized vanishing point and the second hypothesized vanishing point to obtain a third hypothesized vanishing point.

3. The visual SLAM method based on structure lines according to claim 2, wherein The step of classifying the line features based on the first hypothesized vanishing point, the second hypothesized vanishing point, and the third hypothesized vanishing point to obtain structure lines includes: Establish a polar coordinate grid, where the polar coordinate grid is a polar coordinate grid divided according to preset longitude and latitude in a polar coordinate system; For all line segment pairs on the target image, determine the longitude and latitude of the intersection point of the line segment pairs; Calculate the grid response value based on the longitude and latitude, and update the corresponding polar coordinate grid based on the grid response value; Estimate the corresponding longitude and latitude and the updated polar coordinate grid for each hypothesized vanishing point in each combination of hypothesized vanishing points, and screen out the best combination of vanishing points; Classify the line features based on the best combination of vanishing points to obtain structure lines.

4. The visual SLAM method based on structure lines according to claim 2, wherein The step of performing a cross product based on the two target line features to obtain a first hypothesized vanishing point includes: Perform a cross product based on the vector representation of the target line features to obtain the first pixel coordinates of the intersection point as the first hypothesized vanishing point.

5. The visual SLAM method based on structure lines according to claim 4, wherein The step of determining the second hypothesized vanishing point based on the outer circle of the orthogonal plane of the first hypothesized vanishing point includes: Given the optical center and focal length, convert the pixel coordinates of the first hypothesized vanishing point to an equivalent spherical coordinate system, and determine the longitude and latitude representation of the first hypothesized vanishing point according to the spherical coordinate system; Based on the outer circle of the orthogonal plane of the first hypothesized vanishing point, determine the longitude and latitude representation of the second hypothesized vanishing point; Based on the longitude and latitude representation of the second hypothesized vanishing point, determine the second pixel coordinates of the second hypothesized vanishing point.

6. The visual SLAM method based on structure lines according to claim 5, characterized in that The step of performing a cross product based on the first hypothesized vanishing point and the second hypothesized vanishing point to obtain a third hypothesized vanishing point includes: Perform a cross product on the first pixel coordinates of the first hypothesized vanishing point and the second pixel coordinates of the second hypothesized vanishing point to obtain the third pixel coordinates of the third hypothesized vanishing point.

7. A visual SLAM device based on structural lines, characterized in that, The visual SLAM device based on structure lines includes: An acquisition module for acquiring a target image and IMU data; An IMU module for pre-integrating the IMU data according to the IMU motion model to obtain an IMU residual; A determination module, configured to determine each combination of hypothesized vanishing points according to line features in a target image, where the combination of hypothesized vanishing points includes a first hypothesized vanishing point, a second hypothesized vanishing point, and a third hypothesized vanishing point; A classification module, configured to classify the line features based on the first hypothesized vanishing point, the second hypothesized vanishing point, and the third hypothesized vanishing point to obtain structural lines; A pose estimation module, configured to estimate a pose by performing sliding window optimization based on the IMU residuals, the structural lines, and a SLAM algorithm.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, the steps of the visual SLAM method based on structural lines according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, the steps of the visual SLAM method based on structural lines according to any one of claims 1 to 6 are implemented.

Images