Map construction method and device based on visual SLAM

Abstract

The invention relates to a map construction method and device based on visual SLAM (Simultaneous Localization and Mapping). The method comprises the following steps: acquiring a target environment image in real time through a camera; calculating a camera pose according to the target environment image; extracting point, line and surface features in the target environment image, and constructing an error function of similar features and heterogeneous features according to the camera pose; constructing an optimization solver comprising a nonlinear equation and an iterative algorithm, wherein the nonlinear equation is used for calculating the moving trajectory of the camera; inputting the point, line and surface features of the current frame, determining the camera pose of the camera in the next frame relative to the point, line and surface features, and determining the mapping relation of the point, line and surface features of the current frame in the next frame according to the error function; according to the method, the point, line and surface features in the next frame are iteratively optimized according to the mapping relation to complete the construction of the map, and by constructing an error function and iteratively optimizing, the features of different image frames can be effectively matched, mismatching is reduced, and the matching accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of map construction, and in particular to a map construction method and device based on visual SLAM. Background Art

[0002] SLAM (Simultaneous Localization and Mapping), or simultaneous positioning and mapping technology, refers to the operation of smart devices in unknown environments, obtaining three-dimensional or two-dimensional spatial structure information of the surrounding environment through sensors, and simultaneously achieving self-positioning and mapping of the surrounding scene. Visual SLAM is based on the SLAM framework and uses visual sensors (such as monocular cameras, binocular cameras, RGB-D depth cameras, etc.) to build a three-dimensional map of large-scale scenes while positioning itself. Traditional visual SLAM methods include tracking detection, back-end optimization, closed-loop detection, and building maps. Figure 4 There are three parts, among which tracking detection is to roughly estimate the current posture of the camera through the sensor input image; local optimization is to perform global consistent optimization on the posture estimation after tracking detection to eliminate noise interference; closed-loop detection is to optimize the posture through constraints when the camera returns to the original position, so as to make the positioning and posture estimation more accurate; finally, the camera posture and motion obtained above are integrated to create a three-dimensional map of the target environment. However, the performance of traditional SLAM systems is poor in dynamic environments. For example, it is difficult to analyze and construct images in real time during their own movement.

[0003] To this end, the present invention proposes a map construction method and device based on visual SLAM. Summary of the invention

[0004] The present invention aims at the technical problems existing in the prior art and provides a map construction method and device based on visual SLAM.

[0005] The technical solution of the present invention to solve the above technical problems is as follows:

[0006] In a first aspect, the present application provides a map construction method based on visual SLAM, comprising the following steps:

[0007] S1: collect target environment images in real time through the camera;

[0008] S2: Calculate the camera pose according to the target environment image;

[0009] S3: extracting point, line and surface features in the target environment image, and constructing error functions of similar features and heterogeneous features according to the camera pose;

[0010] S4: constructing an optimization solver including a nonlinear equation and an iterative algorithm, wherein the nonlinear equation is used to calculate the trajectory of the camera;

[0011] S5: input the point, line and surface features of the current frame, determine the camera pose of the camera in the next frame relative to the point, line and surface features, and determine the mapping relationship of the point, line and surface features of the current frame in the next frame according to the error function;

[0012] S6: Iteratively optimize the point, line and surface features in the next frame according to the mapping relationship to complete the map construction.

[0013] Furthermore, the map construction method based on visual SLAM, S2 includes:

[0014] The same type of features include: point-to-point feature information, line-to-line feature information and face-to-face feature information;

[0015] The heterogeneous features include: feature information of point to line, feature information of point to surface, and feature information of line to surface;

[0016] The error function includes: for the real position and projection position of feature point i in a frame image, g ij and The reprojection error is:

[0017]

[0018] The distance error from point to line or surface uses the shortest distance from point to line or the shortest distance from point to surface;

[0019] In summary:

[0020] W=∑ i Σ j ||w ij || 2 +Σ o ||w o || 2 ;

[0021] Among them, i represents feature points, j represents camera frames, and o represents other geometric features.

[0022] Furthermore, in the map construction method based on visual SLAM, the process of extracting point, line and surface features in S2 includes:

[0023] The feature points are extracted by the ORB algorithm, and the candidate points in the image are detected by the FAST corner detection algorithm;

[0024] Construct a Gaussian pyramid and detect the scale invariance of feature points in each layer of the pyramid image;

[0025] Calculate the direction of the feature point and convert the feature point into a feature descriptor.

[0026] Furthermore, the map construction method based on visual SLAM calculates feature points for candidate points:

[0027] R=detM-α(traceM) 2 ;

[0028] Among them, detM is the determinant of the matrix M, traceM is the trace of the matrix M, and α is a free parameter whose value range is based on the pixel value of the feature point in the image.

[0029] Furthermore, the map construction method based on visual SLAM, extracting lines and surfaces includes:

[0030] Extracting the line includes extracting two endpoints of the line, and representing the feature descriptor of the line according to an average value of the feature descriptors of the two endpoints;

[0031] Feature extraction for a surface includes extracting the surface according to an image segmentation algorithm and regarding the surface as a feature point with a range.

[0032] Furthermore, in the map construction method based on visual SLAM, the nonlinear equation includes:

[0033] z i (k+1)=h(x(k+1),X i )+v(k+1);

[0034] Among them, h(x(k+1),x i ) is the trajectory change function, which is usually nonlinear, v(k+1) is the change noise, which shows the uncertainty in the trajectory change process, X i =(x i ,y i ) is the coordinate of feature i, z i (k+1) is the camera pose relative to feature point i at time k+1;

[0035] According to the changing distance l and changing angle q between the camera and the feature point, that is, z[l,q] t , derive the formula:

[0036]

[0037] Where θ(k) is the initial angle; the formula is linearized to calculate the Jacobian matrix of the feature point in the camera pose:

[0038]

[0039] Furthermore, the iterative optimization process of the map construction method based on visual SLAM includes:

[0040] Define the error of the feature descriptor mapping in the current frame on the next frame image as:

[0041]

[0042] T a,b is the relative change matrix between the two descriptors, T aw is the camera pose corresponding to the feature descriptor of the current frame, is the change between the feature descriptors a and b; the cost function is obtained by adding the errors of all feature descriptors in the image:

[0043] C = ∑(e a,b p a,b );

[0044] where p a,b is the information matrix of the feature descriptor.

[0045] On the other hand, the present application also provides a map construction device based on visual SLAM, wherein the map construction device is used to implement any of the map construction methods based on visual SLAM, and the map construction device comprises:

[0046] Acquisition unit: collects target environment images in real time through the camera;

[0047] Calculation unit: calculates the camera pose according to the target environment image;

[0048] Extraction unit: extracting point, line and surface features in the target environment image, and constructing error functions of similar features and heterogeneous features according to the camera posture;

[0049] Construction unit: construct an optimization solver including a nonlinear equation and an iterative algorithm, wherein the nonlinear equation is used to calculate the running trajectory of the camera;

[0050] Solving unit: input the point, line and surface features of the current frame, determine the camera pose relative to the point, line and surface features in the next frame, and determine the mapping relationship of the point, line and surface features of the current frame in the next frame according to the error function;

[0051] Iteration unit: Iteratively optimizes the point, line and surface features in the next frame according to the mapping relationship to complete the map construction.

[0052] At the same time, the present application also provides an electronic device, including:

[0053] Memory for storing computer software programs;

[0054] A processor is used to read and execute the computer software program, thereby implementing any of the visual SLAM-based map construction methods.

[0055] A non-transitory computer-readable storage medium having a computer software program stored therein, wherein the computer software program, when executed by a processor, implements any one of the visual SLAM-based map construction methods described above.

[0056] The beneficial effects of the present invention are:

[0057] The image data obtained by the camera not only contains rich color and texture information, but also can extract a large number of feature points, lines and surfaces. These features usually have high discrimination and stability in the environment, which helps to improve the positioning and mapping accuracy of the SLAM system. Visual SLAM can use high-resolution image data to build fine-grained maps, including detailed environmental features such as walls and furniture, thereby achieving high-precision environmental modeling;

[0058] Visual SLAM not only extracts point features, but also extracts line features and surface features, providing rich geometric information; the relationship between these features (such as point-to-point, point-to-line, point-to-surface, etc.) can provide additional constraints, which helps to improve the robustness and accuracy of the system; by constructing an error function and performing iterative optimization, visual SLAM can effectively match features between different image frames, reduce mismatches, and improve matching accuracy;

[0059] Visual SLAM can adapt to various types of cameras, including monocular cameras, stereo cameras, depth cameras, etc., and select the most suitable hardware equipment according to different application scenarios. Visual SLAM utilizes the rich information in the image and can maintain high robustness and reliability under complex conditions such as lighting changes, dynamic environments, partial occlusions, etc. Through nonlinear optimization and iterative algorithms, visual SLAM can minimize errors globally and ensure the overall consistency of maps and pose estimates. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a flowchart of a map construction method based on visual SLAM of the present invention;

[0061] Figure 2 It is a schematic diagram of a map construction device based on visual SLAM of the present invention;

[0062] Figure 3 A schematic diagram of an electronic device according to an embodiment of the present invention;

[0063] Figure 4A schematic diagram of an embodiment of a computer-readable storage medium provided in an embodiment of the present invention.

[0064] In the accompanying drawings, the components represented by the reference numerals are listed as follows:

[0065] Electronic device 500 , memory 510 , processor 520 , computer program 511 ; computer readable storage medium 600 , computer program 611 . DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0067] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0068] In the description of the present application, the term "for example" is used to mean "used as an example, illustration or description". Any embodiment described as "for example" in the present application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid unnecessary details to obscure the description of the present invention. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.

[0069] Example 1

[0070] The present application provides a map construction method based on visual SLAM, comprising the following steps:

[0071] S1: collect target environment images in real time through the camera;

[0072] S2: Calculate the camera pose according to the target environment image;

[0073] S3: extracting point, line and surface features in the target environment image, and constructing error functions of similar features and heterogeneous features according to the camera pose;

[0074] S4: constructing an optimization solver including a nonlinear equation and an iterative algorithm, wherein the nonlinear equation is used to calculate the trajectory of the camera;

[0075] S5: input the point, line and surface features of the current frame, determine the camera pose of the camera in the next frame relative to the point, line and surface features, and determine the mapping relationship of the point, line and surface features of the current frame in the next frame according to the error function;

[0076] S6: Iteratively optimize the point, line and surface features in the next frame according to the mapping relationship to complete the map construction.

[0077] In this embodiment, for example, in an indoor environment, the camera installed on the robot collects the surrounding images in real time, and the specific position and orientation of the camera in the room can be determined by calculating the feature points in these images (such as the corners of the room, the edges of the table, etc.); when extracting features, the ORB algorithm is used to extract the feature points in the image, which may be some unique signs in the room. Then, an error function is constructed, for example, by calculating the position changes of these feature points in different image frames, to accurately calculate the camera pose; an optimization solver is constructed to calculate the trajectory of the camera; this solver needs to consider the changes of the feature points and lines extracted from the image, and optimize the camera trajectory through an iterative algorithm; for example, through multiple calculations and optimizations, the accuracy of the trajectory of the camera during movement is ensured; finally, by optimizing the feature points extracted from each frame of the image, a map of the entire environment is constructed frame by frame; this map can help the robot navigate or be used for scene construction in augmented reality applications; this method realizes high-precision calculation of the camera pose and construction of the environment map through precise feature extraction and optimization algorithms, thereby improving the robustness and accuracy of the SLAM system.

[0078] In one embodiment, S2 includes:

[0079] The similar features include: point-to-point feature information, line-to-line feature information and face-to-face feature information; point-to-point, line-to-line and face-to-face feature information can directly reflect the position and direction changes of the camera in different image frames; through the matching and error calculation of these features, the camera's posture can be accurately estimated

[0080] The heterogeneous features include: feature information of point to line, feature information of point to plane, and feature information of line to plane; the feature information of point to line, point to plane, and line to plane provide additional constraints, further improving the accuracy and robustness of camera pose estimation;

[0081] In some embodiments, in the point-to-point feature information, it is assumed that a feature point A is detected in the first frame image with coordinates (100, 120), and a corresponding feature point B is detected in the second frame image with coordinates (102, 118); repeated detection of point A and point B is used to calculate the relative movement and error of the camera; in the line-to-line feature information, it is assumed that a line L1 is detected in the first frame image with endpoints (50, 50) and (150, 150), and a corresponding line L2 is detected in the second frame image with endpoints (50, 50, 150). (55,48) and (153,148), by comparing the position changes of the line segment endpoints, the camera's position change is determined; in the face-to-face feature information, it is assumed that the wall F1 is detected in the first frame image, and its boundary coordinates are a rectangle [(30,30), (30,130), (130,130), (130,30)], and the corresponding wall F2 is detected in the second frame image, and its boundary coordinates are [(32,28), (32,128), (134,128), (134,28)]. By comparing the changes of these two faces, the camera's movement trajectory can be inferred.

[0082] At the same time, the error function includes: for the real position and projection position of feature point i in a frame image, g ij and The reprojection error is:

[0083] By calculating the error between the true position and the projected position of the feature points in different image frames, errors in feature matching can be identified and corrected;

[0084] The distance error from point to line or surface uses the shortest distance from point to line or the shortest distance from point to surface. The distance error from point to line or point to surface provides geometric constraints for feature matching, which helps to remove mismatched feature points and further improve the matching accuracy.

[0085] In summary:

[0086] W=∑ i Σ j ||w ij || 2 +Σ o ||w o || 2 ;

[0087] Among them, i represents feature points, j represents camera frames, and o represents other geometric features.

[0088] In some embodiments, assuming that the real position of the feature point P in a certain image frame is (210, 260) and the projected position is (208, 258), then the reprojection error is: (210-208, 260-258) = (2, 2), the distance error from point to line or point to surface: the shortest distance from point P to line segment L is 5 pixels, and the shortest distance to the surface is 10 pixels. The above two errors are weighted and combined to minimize the error in the optimization solver.

[0089] The process of extracting point, line and surface features in step S2 includes:

[0090] The feature points are extracted by the ORB algorithm, and the candidate points in the image are detected by the FAST corner detection algorithm;

[0091] Construct a Gaussian pyramid and detect the scale invariance of feature points in each layer of the pyramid image;

[0092] Calculate the direction of the feature point and convert the feature point into a feature descriptor.

[0093] Calculate feature points for candidate points:

[0094] R=detM-α(traceM) 2 ;

[0095] Among them, detM is the determinant of the matrix M, traceM is the trace of the matrix M, and α is a free parameter whose value range is based on the pixel value of the feature point in the image.

[0096] Extraction of lines and surfaces includes:

[0097] Extracting the line includes extracting two endpoints of the line, and representing the feature descriptor of the line according to an average value of the feature descriptors of the two endpoints;

[0098] Feature extraction for a surface includes extracting the surface according to an image segmentation algorithm and regarding the surface as a feature point with a range.

[0099] In some embodiments, ORB (Oriented FAST and Rotated BRIEF) is a commonly used feature detection and description method; FAST (Features from Accelerated Segment Test) corner detection algorithm is used to quickly detect corners in an image;

[0100] Specific process: Detect candidate points: Use the FAST algorithm to find points with drastic brightness changes (i.e. corner points) in the image.

[0101] For example: suppose that in an image, FAST detects multiple corner points, such as the edge of a table, the corner of a bookshelf, etc.

[0102] Gaussian pyramid construction: Gaussian blur the original image and downsample it layer by layer to construct image pyramids of different scales. Scale invariance detection: detect feature points in each layer of the pyramid image to ensure that these feature points are robust at different scales. Assuming the original image is 1000x1000 pixels, construct images of different scales such as 500x500 and 250x250 through the Gaussian pyramid, and detect feature points in each layer of the image.

[0103] Direction calculation: Calculate the gradient direction of the pixels around the feature point to determine the main direction of the feature point.

[0104] Generate descriptors: Use the BRIEF descriptor to generate a unique feature vector based on the direction of the feature point; assuming that the direction of the detected feature point is 45 degrees, a 128-dimensional feature vector is generated through the BRIEF descriptor for feature matching

[0105] Calculate the matrix M: construct the matrix M based on the pixel values ​​around the feature points;

[0106] Calculate the determinant and trace: Calculate the determinant (detM) and trace (traceM) of the matrix M;

[0107] Calculate R value: Use the formula to calculate the eigenvalue R of the candidate point.

[0108] Assuming that the determinant of matrix M is 12000 and the trace is 300, the characteristic calculation formula is: [R = \frac{12000}{300} = 40]

[0109] Extract line and surface features

[0110] Extract line segments: detect line segments in the image and extract their two endpoints; calculate the average descriptor: calculate the feature descriptor of the line segment based on the feature descriptors of the endpoints. Suppose a line segment is detected with endpoints (100, 150) and (200, 250), and the feature descriptors of the endpoints are (0.6, 0.7) and (0.8, 0.9) respectively. Then the feature descriptor of the line segment is the average value of the endpoint descriptors (0.7, 0.8).

[0111] Extracting faces: Face features are detected by image segmentation algorithms and their features are extracted.

[0112] Image segmentation: Use image segmentation algorithm to detect surface features in the image and convert them into feature points: Convert the detected surface features into feature points with ranges. Assume that a rectangular area is detected by the image segmentation algorithm, with the center point (80,80) and the width and height of the area being 100 and 100 pixels respectively. Represent this surface feature as the center point coordinates and its range.

[0113] In one embodiment, the nonlinear equation comprises:

[0114] z i (k+1)=h(x(k+1),X i )+v(k+1);

[0115] Among them, h(x(k+1),x i ) is the trajectory change function, which is usually nonlinear, v(k+1) is the change noise, which shows the uncertainty in the trajectory change process, X i =(x i ,y i ) is the coordinate of feature i, z i (k+1) is the camera pose relative to feature point i at time k+1;

[0116] According to the changing distance l and changing angle q between the camera and the feature point, that is, z[l,q] t , derive the formula:

[0117]

[0118] Where θ(k) is the initial angle; the formula is linearized to calculate the Jacobian matrix of the feature point in the camera pose:

[0119]

[0120] The iterative optimization process includes:

[0121] Define the error of the feature descriptor mapping in the current frame on the next frame image as:

[0122]

[0123] T a,b is the relative change matrix between the two descriptors, T aw is the camera pose corresponding to the feature descriptor of the current frame, is the change between the feature descriptors a and b; the cost function is obtained by adding the errors of all feature descriptors in the image:

[0124] C = ∑(e a,b p a,b );

[0125] where p a,b It is the information matrix of the feature descriptor. Through iterative optimization, the cost function is minimized to accurately estimate the mapping relationship between the camera's pose change and the feature points.

[0126] Through the linearization and iterative optimization of the above nonlinear equations, high-precision camera pose estimation and environment map construction can be achieved. The nonlinear equation takes into account the nonlinear characteristics of camera trajectory changes, and iterative optimization minimizes the error function and continuously adjusts the camera pose and feature point positions, making map construction more accurate and robust.

[0127] Example 2

[0128] The present application also provides a map construction device based on visual SLAM, the map construction device is used to implement any of the map construction methods based on visual SLAM, the map construction device comprises:

[0129] Acquisition unit: collects target environment images in real time through the camera;

[0130] Calculation unit: calculates the camera pose according to the target environment image;

[0131] Extraction unit: extracting point, line and surface features in the target environment image, and constructing error functions of similar features and heterogeneous features according to the camera posture;

[0132] Construction unit: construct an optimization solver including a nonlinear equation and an iterative algorithm, wherein the nonlinear equation is used to calculate the running trajectory of the camera;

[0133] Solving unit: input the point, line and surface features of the current frame, determine the camera pose relative to the point, line and surface features in the next frame, and determine the mapping relationship of the point, line and surface features of the current frame in the next frame according to the error function;

[0134] Iteration unit: Iteratively optimizes the point, line and surface features in the next frame according to the mapping relationship to complete the map construction.

[0135] In this embodiment, different cameras can be used for different application scenarios; for example: the robot uses a monocular camera to construct a map of the indoor environment, extracts feature points in the image through algorithms such as ORB and FAST, and estimates the camera pose. The use of a monocular camera requires relying on feature extraction and matching algorithms to estimate depth information, and the calculation complexity is relatively high; a stereo camera is used for environmental perception and obstacle detection of unmanned vehicles, and directly obtains the depth information of the road ahead through stereo vision to construct a high-precision 3D map. The stereo camera can directly obtain the depth information of the scene through two parallel mounted lenses; a depth camera (such as Microsoft Kinect) is used for 3D modeling of the indoor environment, and a detailed environmental map is constructed by obtaining the depth information of each point. Regardless of the type of camera, in the method of this application, except for slight differences in the method of extracting points, lines and surfaces, the method can be used for map construction to achieve high-precision map construction in different application scenarios.

[0136] See also Figure 3 , Figure 3 Schematic diagram of an electronic device provided by an embodiment of the present invention. Figure 3 As shown, an embodiment of the present invention provides an electronic device 500, including a memory 510, a processor 520, and a computer program 511 stored in the memory 510 and executable on the processor 520. When the processor 520 executes the computer program 511, the following steps are implemented:

[0137] S1: collect target environment images in real time through the camera;

[0138] S2: Calculate the camera pose according to the target environment image;

[0139] S3: extracting point, line and surface features in the target environment image, and constructing error functions of similar features and heterogeneous features according to the camera pose;

[0140] S4: constructing an optimization solver including a nonlinear equation and an iterative algorithm, wherein the nonlinear equation is used to calculate the trajectory of the camera;

[0141] S5: input the point, line and surface features of the current frame, determine the camera pose of the camera in the next frame relative to the point, line and surface features, and determine the mapping relationship of the point, line and surface features of the current frame in the next frame according to the error function;

[0142] S6: Iteratively optimize the point, line and surface features in the next frame according to the mapping relationship to complete the map construction.

[0143] See also Figure 4 , Figure 4 A schematic diagram of an embodiment of a computer-readable storage medium provided in an embodiment of the present invention. Figure 4 As shown, this embodiment provides a computer-readable storage medium 600, on which a computer program 611 is stored. When the computer program 611 is executed by a processor, the following steps are implemented:

[0144] S1: collect target environment images in real time through the camera;

[0145] S2: Calculate the camera pose according to the target environment image;

[0146] S3: extracting point, line and surface features in the target environment image, and constructing error functions of similar features and heterogeneous features according to the camera pose;

[0147] S4: constructing an optimization solver including a nonlinear equation and an iterative algorithm, wherein the nonlinear equation is used to calculate the trajectory of the camera;

[0148] S5: input the point, line and surface features of the current frame, determine the camera pose of the camera in the next frame relative to the point, line and surface features, and determine the mapping relationship of the point, line and surface features of the current frame in the next frame according to the error function;

[0149] S6: Iteratively optimize the point, line and surface features in the next frame according to the mapping relationship to complete the map construction.

[0150] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0151] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0152] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0153] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0154] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0155] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0156] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A map construction method based on visual SLAM, characterized in that: The following steps are involved: S1: collect target environment images in real time through the camera; S2: Calculate the camera pose according to the target environment image; S3: extracting point, line and surface features in the target environment image, and constructing error functions of similar features and heterogeneous features according to the camera pose; S4: constructing an optimization solver including a nonlinear equation and an iterative algorithm, wherein the nonlinear equation is used to calculate the trajectory of the camera; S5: input the point, line and surface features of the current frame, determine the camera pose of the camera in the next frame relative to the point, line and surface features, and determine the mapping relationship of the point, line and surface features of the current frame in the next frame according to the error function; S6: Iteratively optimize the point, line and surface features in the next frame according to the mapping relationship to complete the map construction.

2. The map construction method based on visual SLAM according to claim 1, characterized in that: The S2 includes: The same type of features include: point-to-point feature information, line-to-line feature information and face-to-face feature information; The heterogeneous features include: feature information of point to line, feature information of point to surface, and feature information of line to surface; The error function includes: for the real position and projection position of feature point i in a frame image, g ij and The reprojection error is: The distance error from point to line or surface uses the shortest distance from point to line or the shortest distance from point to surface; In summary: W=∑ i ∑ j ||w ij || 2 +Σ o ||w o || 2 ; Among them, i represents feature points, j represents camera frames, and o represents other geometric features.

3. The map construction method based on visual SLAM according to claim 2, characterized in that: The process of extracting point, line and surface features in S2 includes: The feature points are extracted by the ORB algorithm, and the candidate points in the image are detected by the FAST corner detection algorithm; Construct a Gaussian pyramid and detect the scale invariance of feature points in each layer of the pyramid image; Calculate the direction of the feature point and convert the feature point into a feature descriptor.

4. The map construction method based on visual SLAM according to claim 3, characterized in that: Calculate feature points for candidate points: R=detM-α(traceM) 2 ; Among them, detM is the determinant of the matrix M, traceM is the trace of the matrix M, and α is a free parameter whose value range is based on the pixel value of the feature point in the image.

5. The map construction method based on visual SLAM according to claim 4, characterized in that: Extraction of lines and surfaces includes: Extracting the line includes extracting two endpoints of the line, and representing the feature descriptor of the line according to an average value of the feature descriptors of the two endpoints; Feature extraction for a surface includes extracting the surface according to an image segmentation algorithm and regarding the surface as a feature point with a range.

6. The map construction method based on visual SLAM according to claim 5, characterized in that: The nonlinear equations include: z i (k+1)=h(x(k+1),X i )+v(k+1); Among them, h(x(k+1),x i ) is the trajectory change function, which is usually nonlinear, v(k+1) is the change noise, which shows the uncertainty in the trajectory change process, X i =(x i ,y i ) is the coordinate of feature i, z i (k+1) is the camera pose relative to feature point i at time k+1; According to the changing distance l and changing angle q between the camera and the feature point, that is, z[l,q] t , derive the formula: Where θ(k) is the initial angle; the formula is linearized to calculate the Jacobian matrix of the feature point in the camera pose:

7. The map construction method based on visual SLAM according to claim 6, characterized in that: The iterative optimization process includes: Define the error of the feature descriptor mapping in the current frame on the next frame image as: T a,b is the relative change matrix between the two descriptors, T aw is the camera pose corresponding to the feature descriptor of the current frame, is the change between the feature descriptors a and b; the cost function is obtained by adding the errors of all feature descriptors in the image: C=Σ(e a,b p a,b ): where p a,b is the information matrix of the feature descriptor.

8. A map construction device based on visual SLAM, the map construction device is used to implement the map construction method based on visual SLAM according to any one of claims 1 to 7, the map construction device comprising: Acquisition unit: collects target environment images in real time through the camera; Calculation unit: calculates the camera pose according to the target environment image; Extraction unit: extracting point, line and surface features in the target environment image, and constructing error functions of similar features and heterogeneous features according to the camera posture; Construction unit: construct an optimization solver including a nonlinear equation and an iterative algorithm, wherein the nonlinear equation is used to calculate the running trajectory of the camera; Solving unit: input the point, line and surface features of the current frame, determine the camera pose relative to the point, line and surface features in the next frame, and determine the mapping relationship of the point, line and surface features of the current frame in the next frame according to the error function; Iteration unit: Iteratively optimizes the point, line and surface features in the next frame according to the mapping relationship to complete the map construction.

9. An electronic device, characterized in that: include: Memory for storing computer software programs; A processor is used to read and execute the computer software program, thereby implementing the map construction method based on visual SLAM as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium, characterized in that: The storage medium stores a computer software program, and when the computer software program is executed by the processor, the map construction method based on visual SLAM as described in any one of claims 1 to 7 is implemented.

Images