Indoor positioning method, device, equipment and storage medium based on multi-visual feature fusion

Abstract

The present invention relates to an indoor positioning method using multi-visual feature fusion, which is applied to a robot carrying an RGB-D camera. The method comprises the following steps: obtaining each frame of image captured by the RGB-D camera and performing preprocessing; obtaining point features and point descriptors, line features and line descriptors, and surface features of each frame; obtaining point feature matching relationships, line feature matching relationships, and surface feature matching relationships between images of different frames; constructing a reprojection error function of the point features, line features, and surface features; constructing a joint error function based on the reprojection error function, and performing LM algorithm processing on the joint error function to obtain the pose transformation between the target image of the current frame and the reference frame. The present application uses point features extracted based on deep learning and fuses them with line features and surface features, effectively improving the accuracy of indoor positioning.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an indoor positioning method, device, equipment and storage medium for multi-visual feature fusion. Background Art

[0002] The SLAM positioning algorithm based on visual features uses camera sensors to collect image data. Compared with laser sensors, cameras are not only small in size, light in weight, low in power consumption, and low in price, but the representation of image data is closer to human eye habits and can provide rich feature data. Currently, visual SLAM positioning has gradually become the mainstream direction of SLAM positioning.

[0003] Existing indoor SLAM positioning algorithms based on visual features in indoor environments are mostly based on man-made scenes, such as window sills and floors. These low-texture areas have a low positioning success rate. Summary of the Invention

[0004] Based on this, the present invention provides a method, device, equipment, and storage medium for indoor positioning using multi-visual feature fusion. This method effectively improves the accuracy of indoor positioning by fusing point features with line features and surface features based on deep learning.

[0005] According to a first aspect of some embodiments of the present application, a multi-visual feature fusion indoor positioning method is provided, which is applied to a robot carrying an RGB-D camera, and the method includes the following steps:

[0006] Obtain each frame of image captured by the RGB-D camera and perform preprocessing;

[0007] Input each frame of the preprocessed image into the SuperPoint network to obtain the point features and point descriptors of each frame of the image, perform LSD algorithm processing on each frame of the preprocessed image to obtain the line features and line descriptors of each frame of the image, and perform CAPE algorithm processing on each frame of the preprocessed image to obtain the surface features of each frame of the image;

[0008] Calculating the distance between point descriptors of inter-frame images, and obtaining the point feature matching relationship of the inter-frame images based on the distance; calculating the distance between line descriptors of inter-frame images, and obtaining the line feature matching relationship of the inter-frame images based on the distance; calculating the distance and angle between surface features of inter-frame images, and obtaining the surface feature matching relationship of the inter-frame images, wherein the surface features of the inter-frame images are expressed in a parameterized form of Hassion;

[0009] Selecting the point feature matching relationship between the current frame and the reference frame to construct a reprojection error function of the point feature, selecting the line feature matching relationship between the current frame and the reference frame to construct a reprojection error function of the line feature, and selecting the surface feature matching relationship between the current frame and the reference frame to construct an error function of the surface feature;

[0010] A joint error function is constructed based on the reprojection error function of the point feature, the reprojection error function of the line feature and the error function of the surface feature, and the joint error function is processed by the LM algorithm to obtain the posture transformation between the current frame and the target image of the reference frame.

[0011] According to a second aspect of some embodiments of the present application, a device for indoor positioning using multi-visual feature fusion is provided, comprising:

[0012] A data processing module is used to obtain each frame of image captured by the RGB-D camera and perform preprocessing;

[0013] A feature extraction module is used to input each frame of the preprocessed image into the SuperPoint network to obtain the point features and point descriptors of each frame of the preprocessed image, perform LSD algorithm processing on each frame of the preprocessed image to obtain the line features and line descriptors of the preprocessed image, and perform CAPE algorithm processing on each frame of the preprocessed image to obtain the surface features of the preprocessed image;

[0014] A feature matching module is configured to calculate the distance between point descriptors of inter-frame images, and obtain the point feature matching relationship of the different frames based on the distance; calculate the distance between line descriptors of inter-frame images, and obtain the line feature matching relationship of the different frames based on the distance; calculate the distance and angle between surface features of inter-frame images, and obtain the surface feature matching relationship of the inter-frame images, wherein the surface features of the inter-frame images are expressed in a parameterized form of Hassion;

[0015] a data association module, configured to select the point feature matching relationship between the current frame and the reference frame to construct a reprojection error function of the point feature, select the line feature matching relationship between the current frame and the reference frame to construct a reprojection error function of the line feature, and select the surface feature matching relationship between the current frame and the reference frame to construct an error function of the surface feature;

[0016] The pose solving module is used to construct a joint error function based on the reprojection error function of the point feature, the reprojection error function of the line feature and the error function of the surface feature, and perform LM algorithm processing on the joint error function to solve the pose transformation between the current frame and the target image of the reference frame.

[0017] According to a third aspect of some embodiments of the present application, there is provided a device, including:

[0018] at least one memory and at least one processor;

[0019] The memory is used to store one or more programs;

[0020] When the one or more programs are executed by the at least one processor, the at least one processor implements the steps of the indoor positioning method of multi-visual feature fusion as described in any one of the first aspects.

[0021] According to a fourth aspect of some embodiments of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method as described in any one of the first aspects are implemented.

[0022] This application is based on the fusion of multiple features such as points, lines and surfaces, which avoids the problem of positioning failure in indoor low-texture areas due to insufficient number of single feature extractions. The point features extracted based on deep learning are fused with line features and surface features respectively, which effectively improves the accuracy of indoor positioning. And by improving the LSD algorithm used for line feature extraction of images, the scattered line segments that originally belonged to a line segment are merged, and short line segments are eliminated, thereby improving the robustness of line features and improving indoor positioning accuracy. Finally, this application uses the TUM public indoor low-texture dataset for comparative testing, using absolute trajectory error (ATE) as the evaluation criterion. The results show that the algorithm based on single-point features loses tracking and fails to locate in this environment. Compared with the existing point-line-surface feature fusion algorithm, the positioning accuracy is effectively improved by 21%.

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a flow chart of an indoor positioning method using multi-visual feature fusion in an embodiment of the present application;

[0025] Figure 2 This is a schematic diagram of the steps of a multi-visual feature fusion indoor positioning method in an embodiment of the present application;

[0026] Figure 3 Schematic diagram of reprojection of line features of an indoor positioning device using multi-visual feature fusion in an embodiment of the present application;

[0027] Figure 4 This is a structural diagram of an indoor positioning device that integrates multiple visual features in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the objectives, technical solutions and advantages of the present application clearer, the embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

[0029] It should be clear that the embodiments described are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the embodiments of the present application.

[0030] The terms used in the embodiments of the present application are for the purpose of describing specific embodiments only and are not intended to limit the embodiments of the present application. The singular forms "a," "the," and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0031] When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims. In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.

[0032] In addition, in this application, unless otherwise specified, "plurality" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0033] The positioning algorithms based on single point, line, and surface features mentioned in the background technology may encounter positioning failure problems caused by insufficient features.

[0034] See also Figure 1 and Figure 2 The present application provides an indoor positioning method based on multi-visual feature fusion, which is applied to a robot carrying an RGB-D camera. The method comprises the following steps:

[0035] Step S1: Obtain each frame of image captured by the RGB-D camera and perform preprocessing.

[0036] RGB-D cameras, also known as depth cameras, are characterized by their ability to measure an object's distance from the camera by actively emitting light and receiving the return light, similar to laser sensors, using infrared structured light or Time-of-Flight (ToF) principles. Therefore, they provide richer information than traditional cameras and eliminate the time-consuming and labor-intensive depth calculations required by monocular or binocular cameras.

[0037] In a specific embodiment, before obtaining each frame of image captured by the RGB-D camera in step S1, the method further includes:

[0038] The RGB-D camera is calibrated to obtain internal parameters of the RGB-D camera, wherein the internal parameters include the camera focal length and the pixel coordinates of the image plane origin. The calibration indicates adjustment of the parameters of the RGB-D camera.

[0039] In a specific embodiment, obtaining each frame of image captured by the RGB-D camera and performing preprocessing includes:

[0040] Denoising is performed on each frame of the image to obtain an RGB-D image containing a color image and a depth image, and the color image is converted into a grayscale image.

[0041] Step S2: Input each frame of the preprocessed image into the SuperPoint network to obtain the point features and point descriptors of each frame of the image, perform LSD algorithm processing on each frame of the preprocessed image to obtain the line features and line descriptors of each frame of the image, and perform CAPE algorithm processing on each frame of the preprocessed image to obtain the surface features of each frame of the image.

[0042] The point features, line features and surface features of the pre-processed images of different frames captured by the camera are obtained respectively, and all features of the image are fully obtained to ensure the integrity of the data in subsequent processing.

[0043] Point features consist of key points and descriptors. A key point refers to the location of the feature point in the image. It is a representative pixel point whose pixel value is significantly different from the pixel values ​​of its surrounding pixels. Descriptors are designed based on the principle that features with similar appearance should have similar descriptors. They are generally composed of multi-dimensional 0,1 vectors. In theory, the descriptor of the same feature point in different images is also the same. SuperPoint is an advanced self-supervised framework for feature point detection and descriptor calculation. It consists of two parts: MagicPoint and Homographic. The former is a VGG neural network-based framework for feature point detection, and the latter is a descriptor for generating feature points. SuperPoint's robustness to changes in ambient lighting and viewpoint makes it more stable.

[0044] CAPE is a fast plane extraction algorithm proposed by Proenca and Gao. It divides the point cloud into plane networks, calculates the model parameters of each network through principal component analysis (PCA), and then uses the plane normal vector to perform histogram statistics, thereby completing the plane segmentation by region growing of each grid. Compared with existing plane extraction methods, the CAPE method is faster and the plane segmentation results are more consistent.

[0045] The core idea of ​​the LSD algorithm is to merge pixels with similar gradient directions. The main disadvantages of using LSD to extract line features are: it sets a flag for each pixel to indicate whether it is used in the region growing step, so each pixel belongs to only one straight line. When two straight lines intersect on the image, at least one straight line is divided into two segments. In addition, due to occlusion, the original line segment will be divided into multiple segments. Therefore, in order to avoid the above situation, in a preferred embodiment, each frame of the preprocessed image is processed by the LSD algorithm to obtain the line features and line descriptors of the preprocessed image, including:

[0046] Step S21: Use the LSD algorithm to perform line segment detection on the preprocessed images of different frames, and calculate the gradient value and gradient direction of the line segment, fit a plurality of the line features and form a line feature set.

[0047] Step S22: Obtain the first angle between any two line segments in the line feature set of each frame image, the second angle between the straight line formed by the midpoints of the two line segments and the two line segments, and the minimum distance between the endpoints of the two line segments.

[0048] Step S23: When the first angle and the second angle are less than 1° and the minimum distance is less than three pixels, determine that the two line segments are merged into one line segment.

[0049] Step S24: Eliminate line segments with lengths less than five pixels from the line feature set of each frame of image, and obtain line features and line descriptors of the pre-processed images of different frames.

[0050] Step S3: Calculate the distance between the point descriptors of the inter-frame images, and obtain the point feature matching relationship of the different frames according to the distance; calculate the distance between the line descriptors of the inter-frame images, and obtain the line feature matching relationship of the inter-frame images according to the distance; express the surface features of the inter-frame images in the form of Hassion parameterization, calculate the distance and angle between the surface features of the inter-frame images, and obtain the surface feature matching relationship of the inter-frame images, wherein the surface features of the inter-frame images are expressed in the form of Hassion parameterization.

[0051] Preprocessed inter-frame images have different point features, line features, and surface features. To confirm the positional relationship between inter-frame images, we can express it through feature matching relationships between different frames. The higher the degree of matching between point-to-point features, line-to-line features, and surface-to-surface features, the closer the positional relationship between the corresponding preprocessed images.

[0052] Matching between point features and line features is typically accomplished by calculating the distance between corresponding descriptors. Descriptor distance represents the degree of similarity between two features. The smaller the distance between descriptors, the closer the similarity, and the closer the matching relationship.

[0053] Since face features do not have descriptors, it is not possible to calculate the descriptor distance to complete face feature matching. Usually, they are expressed in a parameterized form of Hassion, and the matching relationship is closely related to the parameter expression.

[0054] In a specific embodiment, step S3: calculating the distance between point descriptors of inter-frame images, and obtaining the point feature matching relationship of the different frames according to the distance, includes:

[0055] The KNN algorithm is used to calculate the distance between the point descriptors of the inter-frame images, and the RANSAC algorithm is used to eliminate the error distance to obtain the matching relationship of the point features of the inter-frame images, wherein the distance is described by the Hamming distance.

[0056] Calculating the distance between line descriptors of inter-frame images, and obtaining the line feature matching relationship of the different frames according to the distance, including:

[0057] The KNN algorithm is used to calculate the distances between the line descriptors in different frames, and the RANSAC algorithm is used to eliminate the error distances to obtain the matching relationship between the line features in different frames, wherein the distance is described by the Hamming distance.

[0058] Calculate the distance and angle between the surface features of the inter-frame images to obtain the surface feature matching relationship between the inter-frame images, where the surface features of the inter-frame images are expressed in the form of Hassion parameterization, including;

[0059] The surface feature is recorded as π=(n T ,d),n=(n x ,n y ,n z ) T is the unit normal vector of the plane where the face feature in the image lies, and d (d>0) is the distance from the camera origin to the plane;

[0060] Obtaining the plane normal vector angle of the plane where the surface feature in the two preprocessed images is located, and the first distance and the second distance from the camera origin to the two planes;

[0061] When the included angle of the plane normal vector is less than 10° and the difference between the first distance and the second distance is less than 0.1 meter, it is determined that a matching relationship is established between the surface features of the two frames.

[0062] Step S4: Select the point feature matching relationship between the current frame and the reference frame to construct a reprojection error function for the point feature. Select the line feature matching relationship between the current frame and the reference frame to construct a reprojection error function for the line feature. Select the surface feature matching relationship between the current frame and the reference frame to construct an error function for the surface feature.

[0063] In most cases, the reference frame is the frame before the current frame. However, if the current frame and the previous frame do not match, the keyframe closest to the current frame is used as the reference frame for matching. The keyframe selection criteria refer to ORBSLAM2.

[0064] The construction of the reprojection error function of the point feature specifically includes:

[0065] Step S411: Obtain the three-dimensional coordinate points of the reference frame using the following formula:

[0066] sp k-1 =KP, where P is the spatial three-dimensional coordinate point of the reference frame, K is the internal parameter of the camera, s is the depth information of the pre-processed image, and p k-1 is the pixel coordinate of the reference frame;

[0067] Step S412: Obtain the spatial three-dimensional coordinate point of the current frame, using the following formula:

[0068] sp k =K(RP+t), where p kis the pixel coordinate of the current frame with the same pixel coordinate as the reference frame, R is the rotation pose transformation between the reference frame and the current frame, and t is the translation pose transformation between the reference frame and the current frame:

[0069] Step S413: The reprojection error function of the point feature is constructed as:

[0070] e p =p k -Π(RP+t), where Π(.) is the projection function related to K,s.

[0071] For easier understanding, see Figure 2 , construct the reprojection error function of the line feature. Specifically, the following steps are included:

[0072] Step S421: Get the space line segment L and use the two end points L1 = (X1, Y1, Z1, r1) T and L2 = (X2, Y2, Z2, r2) T Represents this line segment, denoted by l u1 , l u2 As its non-homogeneous coordinates, the space segment L is expressed in Planck coordinates as:

[0073] Wherein, v is the direction vector of the space line segment L, n is the normal vector of the plane formed by the space line segment L and the coordinate origin, and passes through the coordinate origin;

[0074] Step S422: Project the line segment in the reference frame into three-dimensional space according to the camera's internal parameter K and the depth information s to obtain a spatial line segment L and its two end points Ls and Le, and reproject L into the current frame to obtain a projected line segment l and its two end points l. s 、l e , the formula is as follows:

[0075]

[0076] Step S423: Match the projected line segment l of L to obtain the line segment l′ and the two end points l of the current frame s ′、l e ';

[0077] Step S424: Use l s ′, l e The distance from ′ to the projected line segment l represents the reprojection error of the space line segment L:

[0078] Where d(l′,l) represents the distance between the lines l′ and l

[0079] Constructing the error function of the surface feature, including:

[0080] Step S431: Minimize the parameterization of the face features of the current frame and the reference frame, and express it as:

[0081] where φ and ψ are the azimuth and elevation angles of the normal, n x 、n y 、n z are the unit normal vectors of the x, y, and z axes respectively;

[0082] Step S432: The error function between the plane features of the reference frame and the plane features of the current frame is expressed as:

[0083] in, is the pose transformation from the reference frame to the current frame.

[0084] Step S5: construct a joint error function based on the reprojection error function of the point feature, the reprojection error function of the line feature and the error function of the surface feature, and perform LM algorithm processing on the joint error function to obtain the posture transformation between the current frame and the target image of the reference frame.

[0085] Specifically, the joint error function is:

[0086]

[0087] Among them, P, L, π represent the set of point, line, and surface features respectively, ∑p -1 ,∑l -1 ,∑π -1 Represents the covariance matrix of point, line and surface features respectively, ρ p , ρ l , ρ π Respectively represent the cost values ​​set by the robust Huber kernel function for point, line, and surface features.

[0088] The above problem is a least squares problem. The LM (Levenberg-Marquart) algorithm is used to minimize the joint error function Et. After solving it, the pose transformation between the two frames can be obtained.

[0089] Based on the pose transformation, the positions of different frames are iterated sequentially to achieve positioning. Specifically, the initial position of the robot carrying the camera is known, that is, the position of the first frame is known. The second frame is used as the current frame, and the first frame is used as the reference frame. The pose transformation between the first and second frames is calculated according to the algorithm, and the position of the second frame is obtained. Therefore, once the pose transformation between the current frame and the reference frame is determined, the position of the current frame can be determined. This iterative process allows the position of each frame to be determined, thus achieving positioning.

[0090] Corresponding to the above-mentioned indoor positioning method of multi-visual feature fusion, such as Figure 4 As shown, the present application also provides an indoor positioning device 400 for fusion of multiple visual features, comprising:

[0091] The data processing module 410 is used to obtain each frame of image captured by the RGB-D camera and perform preprocessing.

[0092] The feature extraction module 420 is used to input each frame of the preprocessed image into the SuperPoint network to obtain the point features and point descriptors of each frame of the image, perform LSD algorithm processing on each frame of the preprocessed image to obtain the line features and line descriptors of each frame of the image, and perform CAPE algorithm processing on each frame of the preprocessed image to obtain the surface features of each frame of the image.

[0093] The feature matching module 430 is used to calculate the distance between point descriptors of inter-frame images, and obtain the point feature matching relationship of the inter-frame images based on the distance; calculate the distance between line descriptors of inter-frame images, and obtain the line feature matching relationship of the inter-frame images based on the distance; calculate the distance and angle between surface features of inter-frame images, and obtain the surface feature matching relationship of the inter-frame images, wherein the surface features of the inter-frame images are expressed in a Hassion parameterized form.

[0094] The data association module 440 is used to select the point feature matching relationship between the current frame and the reference frame to construct the reprojection error function of the point feature, select the line feature matching relationship between the current frame and the reference frame to construct the reprojection error function of the line feature, and select the surface feature matching relationship between the current frame and the reference frame to construct the error function of the surface feature.

[0095] The posture solving module 450 is used to construct a joint error function based on the reprojection error function of the point feature, the reprojection error function of the line feature and the error function of the surface feature, and perform LM algorithm processing on the joint error function to solve the posture transformation between the current frame and the target image of the reference frame.

[0096] In an optional example, the feature extraction module 420 includes:

[0097] The line end detection unit is used to perform line segment detection on each frame of image using the LSD algorithm, calculate the gradient value and gradient direction of the line segment, fit a plurality of line features and form a line feature set.

[0098] The line segment selection unit is used to obtain the first angle between any two line segments in the line feature set, the second angle between the straight line formed by the midpoints of the two line segments and the two line segments, and the minimum distance between the endpoints of the two line segments.

[0099] The line segment merging unit is configured to determine that the two line segments are merged into one line segment when the first angle and the second angle are less than 1° and the minimum distance is less than three pixels.

[0100] The line feature extraction unit is used to eliminate line segments with a length of less than five pixels in the line feature set to obtain line features and line descriptors of the preprocessed images of different frames.

[0101] In an optional example, the feature matching module 430 includes:

[0102] A point feature matching unit is used to calculate the distance between the point descriptors of the inter-frame images using the KNN algorithm, and to eliminate the error distance using the RANSAC algorithm to obtain the matching relationship of the point features of the inter-frame images, wherein the distance is described by the Hamming distance;

[0103] A line feature matching unit is used to calculate the distance between the line descriptors of the inter-frame images using the KNN algorithm, and eliminate the error distance using the RANSAC algorithm to obtain the matching relationship of the line features of the inter-frame images, wherein the distance is described by the Hamming distance;

[0104] The surface feature matching unit is used to record the surface feature as π=(n T ,d),n=(n x ,n y ,n z ) T is the unit normal vector of the plane where the surface feature of the inter-frame image is located, and d (d>0) is the distance from the camera origin to the plane; the plane normal vector angle of the plane where the surface feature in the two preprocessed images is located, and the first distance and second distance from the camera origin to the two planes are obtained; when the plane normal vector angle is less than 10° and the difference between the first distance and the second distance is less than 0.1 meter, it is determined that a matching relationship is established between the surface features of the two frames.

[0105] In an optional example, the data association module 440 includes:

[0106] The point feature association unit is used to obtain the spatial three-dimensional coordinate points of the reference frame. The formula is as follows:

[0107] sp k-1 =KP, where P is the spatial three-dimensional coordinate point of the reference frame, K is the internal parameter of the camera, s is the depth information of the preprocessed image, and pk-1 is the pixel coordinate of the reference frame.

[0108] Get the three-dimensional coordinate point of the current frame using the following formula:

[0109] sp k =K(RP+t), where pk is the pixel coordinate of the current frame with the same pixel coordinate as the reference frame, R is the rotational pose transformation between the reference frame and the current frame, and t is the translational pose transformation between the reference frame and the current frame.

[0110] The reprojection error function of the point feature is constructed as:

[0111] e p =p k -Π(RP+t), where Π(.) is the projection function related to K,s.

[0112] Line feature association unit, used to obtain the space line segment L, and use the two end points L1 = (X1, Y1, Z1, r1) T and L2 = (X2, Y2, Z2, r2) T Represent this line segment, record lu1 and lu2 as their non-homogeneous coordinates, and use Planck coordinates to represent the space line segment L as:

[0113] Wherein, v is the direction vector of the space line segment L, n is the normal vector of the plane formed by the space line segment L and the coordinate origin, and passes through the coordinate origin;

[0114] The line segment in the reference frame is projected into the three-dimensional space according to the internal parameter K of the camera and the depth information s to obtain the spatial line segment L and its two end points Ls and Le. L is reprojected into the current frame to obtain the projected line segment l and its two end points l. s 、l e , the formula is as follows:

[0115]

[0116] Match the projected line segment l of L to obtain the line segment l′ and the two end points l of the current frame s ′、l e ';

[0117] Use l s ′, l eThe distance from ′ to the projected line segment l represents the reprojection error of the space line segment L:

[0118] Where d(l′,l) represents the distance between the lines l′ and l;

[0119] The face feature association unit is used to minimize the parameterization of the face feature representation method of the current frame and the reference frame, which is expressed as:

[0120] where φ and ψ are the azimuth and elevation angles of the normal, n x 、n y 、n z are the unit normal vectors of the x, y, and z axes respectively;

[0121] The error function between the plane of the reference frame and the plane of the current frame is expressed as:

[0122] in, is the pose transformation from the reference frame to the current frame.

[0123] In an optional example, the pose solving module 450 includes:

[0124] Solve the pose unit, and the joint error function is:

[0125]

[0126] Among them, P, L, π represent the set of point, line, and surface features respectively, ∑p -1 ,∑l -1 ,∑π -1 Represents the covariance matrix of point, line and surface features respectively, ρ p , ρ l , ρ π Respectively represent the cost values ​​set by the robust Huber kernel function for point, line, and surface features.

[0127] In an optional example, the apparatus 400 further includes:

[0128] The calibration module is used to calibrate the RGB-D camera to obtain internal parameters of the RGB-D camera, including the camera focal length and the pixel coordinates of the image plane origin.

[0129] In an optional example, the apparatus 400 further includes:

[0130] The preprocessing module is used to perform denoising on each frame of the image to obtain an RGB-D image containing a color image and a depth image, and convert the color image into a grayscale image.

[0131] Corresponding to the above-mentioned indoor positioning method of multi-visual feature fusion, the present application also provides a device including at least one memory and at least one processor;

[0132] The memory is used to store one or more programs;

[0133] When the one or more programs are executed by the at least one processor, the at least one processor implements the steps of any one of the above-mentioned methods for indoor positioning using multi-visual feature fusion.

[0134] The implementation process of the functions and effects of each component in the above-mentioned device is specifically detailed in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here. For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely schematic, wherein the components described as separate parts may or may not be physically separated, and the parts displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0135] Corresponding to the above-mentioned indoor positioning method of multi-visual feature fusion, the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, it implements the steps of any of the methods described above.

[0136] The present disclosure may take the form of a computer program product implemented on one or more storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-usable storage media include permanent and non-permanent, removable and non-removable media, and information storage may be implemented by any method or technology. Information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0137] This application is based on the fusion of multiple features such as points, lines and surfaces, which avoids the problem of positioning failure in indoor low-texture areas due to insufficient number of single feature extractions. The point features extracted based on deep learning are fused with line features and surface features respectively, which effectively improves the accuracy of indoor positioning. And by improving the LSD algorithm used for line feature extraction of images, the scattered line segments that originally belonged to a line segment are merged, and short line segments are eliminated, thereby improving the robustness of line features and improving indoor positioning accuracy. Finally, this application uses the TUM public indoor low-texture dataset for comparative testing, using absolute trajectory error (ATE) as the evaluation criterion. The results show that the algorithm based on single-point features loses tracking and fails to locate in this environment. Compared with the existing point-line-surface feature fusion algorithm, the positioning accuracy is effectively improved by 21%.

[0138] It should be understood that the embodiments of the present application are not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the embodiments of the present application is limited only by the appended claims. The embodiments described above only express several implementation methods of the embodiments of the present application, and their descriptions are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the embodiments of the present application, several variations and improvements can be made, which all fall within the scope of protection of the embodiments of the present application.

Claims

1. A multi-visual feature fusion indoor positioning method, applied to a robot carrying an RGB-D camera, characterized in that: The method comprises the following steps: Obtain each frame of image captured by the RGB-D camera and perform preprocessing; Input each frame of the preprocessed image into the SuperPoint network to obtain the point features and point descriptors of each frame of the image, perform LSD algorithm processing on each frame of the preprocessed image to obtain the line features and line descriptors of each frame of the image, and perform CAPE algorithm processing on each frame of the preprocessed image to obtain the surface features of each frame of the image; Calculating the distance between point descriptors of inter-frame images, and obtaining the point feature matching relationship of the inter-frame images based on the distance; calculating the distance between line descriptors of inter-frame images, and obtaining the line feature matching relationship of the inter-frame images based on the distance; calculating the distance and angle between surface features of inter-frame images, and obtaining the surface feature matching relationship of the inter-frame images, wherein the surface features of the inter-frame images are expressed in a parameterized form of Hassion; Selecting the point feature matching relationship between the current frame and the reference frame to construct a reprojection error function of the point feature, selecting the line feature matching relationship between the current frame and the reference frame to construct a reprojection error function of the line feature, and selecting the surface feature matching relationship between the current frame and the reference frame to construct an error function of the surface feature; A joint error function is constructed based on the reprojection error function of the point feature, the reprojection error function of the line feature and the error function of the surface feature, and the joint error function is processed by the LM algorithm to obtain the posture transformation between the target image of the current frame and the reference frame.

2. The indoor positioning method based on multi-visual feature fusion according to claim 1, characterized in that: The line features and line descriptors obtained by performing LSD algorithm processing on each frame of the preprocessed image include: Perform line segment detection on each frame of image using the LSD algorithm, calculate the gradient value and gradient direction of the line segment, and fit multiple line features to form a line feature set; Obtaining a first angle between any two line segments in the line feature set of each frame of image, a second angle between a straight line formed by the midpoints of the two line segments and the two line segments, and a minimum distance between the endpoints of the two line segments; When the first included angle and the second included angle are less than 1° and the minimum distance is less than three pixels, determining that the two line segments are merged into one line segment; Line segments with lengths less than five pixels in the line feature set of each frame of image are eliminated to obtain line features and line descriptors of pre-processed images of different frames.

3. The indoor positioning method based on multi-visual feature fusion according to claim 1, characterized in that: Calculating the distance between point descriptors of inter-frame images, and obtaining a point feature matching relationship of the inter-frame images according to the distance, including: The distances between the point descriptors of the inter-frame images are calculated using the KNN algorithm, and the error distances are eliminated using the RANSAC algorithm to obtain the matching relationship between the point features of the inter-frame images, wherein the distances are described by the Hamming distance; Calculating the distance between line descriptors of inter-frame images, and obtaining a line feature matching relationship between the inter-frame images according to the distance, including: The distances between the line descriptors of the inter-frame images are calculated using the KNN algorithm, and the error distances are eliminated using the RANSAC algorithm to obtain the matching relationship of the line features of the inter-frame images, wherein the distances are described by the Hamming distance; Calculate the distance and angle between the surface features of the inter-frame images to obtain the surface feature matching relationship between the inter-frame images, where the surface features of the inter-frame images are expressed in the form of Hassion parameterization, including; The surface feature is recorded as π=(n T ,d),n=(n x ,n y ,n z ) T is the unit normal vector of the plane where the face feature in the image lies, and d (d>0) is the distance from the camera origin to the plane; Obtaining the plane normal vector angle of the plane where the surface feature is located in the two preprocessed images, and the first distance and the second distance from the camera origin to the two planes; When the included angle of the plane normal vector is less than 10° and the difference between the first distance and the second distance is less than 0.1 meter, it is determined that a matching relationship is established between the surface features of the two frames.

4. The indoor positioning method based on multi-visual feature fusion according to claim 3 is characterized in that: Selecting the point feature matching relationship between the current frame and the reference frame to construct a reprojection error function of the point feature, including: The three-dimensional coordinates of the reference frame are obtained using the following formula: sp k-1 =KP, where P is the spatial three-dimensional coordinate point of the reference frame, K is the internal parameter of the camera, s is the depth information of the pre-processed image, and pk-1 is the pixel coordinate of the reference frame; Get the three-dimensional coordinate point of the current frame using the following formula: sp k =K(RP+t), where pk is the pixel coordinate of the current frame with the same pixel coordinate as the reference frame, R is the rotational pose transformation between the reference frame and the current frame, and t is the translational pose transformation between the reference frame and the current frame: The reprojection error function of the point feature is constructed as: e p =p k -Π(RP+t), where Π(.) is the projection function related to K,s; The selecting the line feature matching relationship between the current frame and the reference frame to construct a reprojection error function of the line feature includes: Get the space line segment L and use the two end points L1 = (X1, Y1, Z1, r1) T and L2 = (X2, Y2, Z2, r2) T Represent this line segment, record lu1 and lu2 as their non-homogeneous coordinates, and use Planck coordinates to represent the space line segment L as: Wherein, v is the direction vector of the space line segment L, n is the normal vector of the plane formed by the space line segment L and the coordinate origin, and passes through the coordinate origin; The line segment in the reference frame is projected into the three-dimensional space according to the internal parameter K of the camera and the depth information s to obtain the spatial line segment L and its two end points Ls and Le. L is reprojected into the current frame to obtain the projected line segment l and its two end points l. s 、l e , the formula is as follows: Match the projected line segment l of L to obtain the line segment l′ and the two end points l of the current frame s ′、l e '; Use l s ′, l e The distance from ′ to the projected line segment l represents the reprojection error of the space line segment L: Where d(l′,l) represents the distance between the lines l′ and l; The selecting a matching relationship between the current frame and the reference frame to construct an error function of the face feature includes: The face features in the current frame and the reference frame are parameterized to a minimum, which is expressed as: where φ and ψ are the azimuth and elevation angles of the normal, n x 、n y 、n z are the unit normal vectors of the x, y, and z axes respectively; The error function between the plane features of the reference frame and the plane features of the current frame is expressed as: in, is the pose transformation from the reference frame to the current frame.

5. The indoor positioning method based on multi-visual feature fusion according to claim 4, characterized in that: The joint error function is: Among them, P, L, π represent the set of point, line, and surface features respectively, ∑p -1 ,∑l -1 ,∑π -1 Represents the covariance matrix of point, line and surface features respectively, ρ p , ρ l , ρ π Respectively represent the cost values ​​set by the robust Huber kernel function for point, line, and surface features.

6. The indoor positioning method based on multi-visual feature fusion according to claim 1, characterized in that: Before acquiring each frame of image captured by the RGB-D camera, the method further includes: The RGB-D camera is calibrated to obtain internal parameters of the RGB-D camera, where the internal parameters include the camera focal length and the pixel coordinates of the image plane origin.

7. The indoor positioning method based on multi-visual feature fusion according to claim 1, characterized in that: Each frame of image captured by the RGB-D camera is obtained and preprocessed, including: Each frame of the image is denoised to obtain an RGB-D image containing a color image and a depth image, and the color image is converted into a grayscale image.

8. A device for indoor positioning based on multi-visual feature fusion, characterized in that: include: The data processing module is used to obtain each frame of image captured by the RGB-D camera and perform preprocessing; A feature extraction module is used to input each frame of the preprocessed image into the SuperPoint network to obtain the point features and point descriptors of each frame of the image, perform LSD algorithm processing on each frame of the preprocessed image to obtain the line features and line descriptors of each frame of the image, and perform CAPE algorithm processing on each frame of the preprocessed image to obtain the surface features of each frame of the image; A feature matching module is configured to calculate the distance between point descriptors of inter-frame images and obtain point feature matching relationships between different frames based on the distance; calculate the distance between line descriptors of inter-frame images and obtain line feature matching relationships between different frames based on the distance; and calculate the distance and angle between the surface features of inter-frame images using the parameterized form of Hassion to obtain surface feature matching relationships between the images, wherein the surface features of the inter-frame images are expressed in the parameterized form of Hassion. a data association module, configured to select the point feature matching relationship between the current frame and the reference frame to construct a reprojection error function of the point feature, select the line feature matching relationship between the current frame and the reference frame to construct a reprojection error function of the line feature, and select the surface feature matching relationship between the current frame and the reference frame to construct an error function of the surface feature; The pose solving module is used to construct a joint error function based on the reprojection error function of the point feature, the reprojection error function of the line feature and the error function of the surface feature, and perform LM algorithm processing on the joint error function to solve the pose transformation between the target image of the current frame and the reference frame.

9. An electronic device for indoor positioning using multi-visual feature fusion, characterized in that: include: at least one memory and at least one processor; The memory is used to store one or more programs; When the one or more programs are executed by the at least one processor, the at least one processor implements the steps of the indoor positioning method based on multi-visual feature fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images