Binocular vision positioning method and system in weak light environment based on deep learning enhancement

Abstract

The application provides a binocular vision positioning method and system in a weak light environment based on deep learning enhancement, and belongs to the technical field of computer vision. The method comprises the following steps: acquiring binocular vision images; performing edge feature enhancement on the images by using an edge detection model; performing line feature extraction on the images after edge feature enhancement by using a straight line detection algorithm, and removing false line segments generated due to point light source scattering by using a light ray erasing logic; extracting point features of the images by using an image point feature extraction model; performing feature matching based on the acquired point features by using an image matching model; screening frames with position, angle change and low feature matching number for backend optimization based on a key frame selection strategy, and outputting a final positioning trajectory; and the backend optimization process comprises optimizing inter-frame poses by constructing a re-projection residual model of point and line features. The feature extraction stability in the weak light environment is improved, the light scattering interference is effectively inhibited, and the practicability is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and particularly relates to a binocular vision positioning method and system in a weak light environment based on deep learning enhancement. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] In recent years, with the rapid development of computer vision and robot technology, the visual SLAM (Simultaneous Localization and Mapping) technology has been widely applied in automatic driving, robot navigation, augmented reality (AR), virtual reality (VR), three-dimensional reconstruction, industrial field and even agricultural field due to its low hardware cost and rich information. The visual SLAM system collects environment images through a camera, extracts feature points or line features, and estimates the camera motion trajectory combined with multiple frame data, so as to realize autonomous positioning and environment modeling.

[0004] The visual SLAM technology is not only affected by the quality of picture data. For example, under weak light conditions, the number of photons received by the camera photosensitive element is reduced, which leads to increased image noise and reduced contrast, and further affects the stability of feature extraction. At the same time, how to ensure the stability of the extracted picture feature points and the stability of the feature matching between picture frames will also greatly affect the accuracy of the back-end optimization. The current visual SLAM front-end feature point extraction and matching methods mainly include geometric-based methods and deep learning-based methods. However, in the case of insufficient light, the positioning result is still not ideal. Low light will introduce noise when extracting feature points for traditional geometric-based methods, and will also introduce noise for deep learning-based feature point extraction methods, resulting in reduced positioning accuracy of the SLAM system based on feature point calculation of inter-frame pose. SUMMARY

[0005] To overcome the above deficiencies of the prior art, the application provides a binocular vision positioning method and system in a weak light environment based on deep learning enhancement, which enhances edge features of original camera images based on an edge detection model (Richer Convolutional Features, RCF) of a convolutional neural network. Secondly, point features are extracted from the original images and matched using an image point feature extraction model such as a SuperPoint model and an image matching model such as a SuperGlue model. Then, line features are extracted using a line detection algorithm (Line Segment Detector, LSD), and light erasing logic is used in this process to reduce the influence of point light source scattering imaging. Finally, key frames are selected to calculate inter-frame poses and use a graph optimization algorithm to obtain the final trajectory.

[0006] To achieve the above object, one or more embodiments of the application provide the following technical solutions:

[0007] The application provides a binocular vision positioning method in a weak light environment based on deep learning enhancement in a first aspect.

[0008] The binocular vision positioning method in a weak light environment based on deep learning enhancement comprises the following steps.

[0009] Obtaining binocular vision images.

[0010] Enhancing edge features of the binocular vision images using an edge detection model.

[0011] Extracting line features from the binocular vision images after edge feature enhancement using a line detection algorithm, and removing false line segments caused by point light source scattering using light erasing logic.

[0012] Extracting point features of the binocular vision images using an image point feature extraction model, matching features based on the extracted point features using a SuperGlue model, judging the positional relationship between the point features and the line features, and using the relationship of point feature matching to obtain the relationship of line feature matching.

[0013] Based on a key frame selection strategy, frames with position, angle changes and low feature matching numbers are screened for backend optimization to output the final positioning trajectory. The backend optimization comprises optimizing inter-frame poses by constructing a re-projection residual model of point and line features.

[0014] As a further technical solution, the edge detection model is based on an improved VGG16 network structure, the fully connected layer and the fifth pooling layer are deleted, and multi-scale feature fusion is used to extract edge information in the image.

[0015] As a further technical solution, a straight line detection algorithm is used to extract line features from the binocular vision image with enhanced edge features, and a light erasing logic is used to remove false line segments caused by point light source scattering, including:

[0016] extracting line features of the binocular vision image with enhanced edge features and geometric center coordinates of light sources in the image;

[0017] extracting pixel values corresponding to endpoints of all lines, and performing contrast stretching on the endpoints of the line segments, and setting a scatter threshold;

[0018] performing gray value calculation on the pixel values of the endpoints after contrast stretching, further excluding line segments whose starting point gray values are all greater than the scatter threshold, and calculating the perpendicular distance from the light source center coordinates to each line, and determining whether it is caused by scatter through a distance threshold.

[0019] As a further technical solution, an image point feature extraction model is used to extract point features of the binocular vision image, including:

[0020] The image point feature extraction model compresses the binocular vision image through an encoder, detects feature point positions of the compressed image, and outputs descriptors;

[0021] The results of the feature point position detection are subjected to a softmax function and a reshape operation to obtain a tensor with the same size as the binocular vision image;

[0022] Select high-confidence feature point information and distribute the feature points evenly on the binocular vision image through grid allocation.

[0023] As a further technical solution, based on the obtained point features, a SuperGlue model is used for feature matching; the positional relationship between the point features and the line features is determined, and the relationship between the line features is obtained using the point feature matching relationship; including:

[0024] Based on the descriptors obtained by the image point feature extraction model, an attention mechanism and a graph neural network are used to optimize feature point matching, and a dustbin mechanism is used to filter incorrect matching pairs;

[0025] After obtaining the matching relationship of the point features, for the point and line features on the same image, it is determined which point features are on the line features or the position distance between the point features and the line features is less than a set matching threshold, then the point features are defined as belonging to the line features, and according to the matching relationship of the point features, the matching relationship of the line features is further obtained.

[0026] As a further technical solution, the process of optimizing the image pose by constructing a re-projection residual model of the point features includes:

[0027] After the point features are extracted and matched, the depth information of the image points is recovered through binocular stereo vision;

[0028] The 3D points in the world coordinate system are transformed into the camera coordinate system;

[0029] The pixel error between the image position of the 3D point after transformation (i.e., re-projection) and the extracted feature point constitutes the re-projection error of the point feature.

[0030] As a further technical solution, the process of constructing the re-projection residual error model of the line feature and optimizing the inter-frame pose includes:

[0031] The depth of the line segment endpoints is calculated for the extracted line features to obtain the information of the spatial 3D line segment.

[0032] The spatial 3D line segment is represented by the Plücker coordinates to obtain the line segment information in the Plücker coordinate system.

[0033] The line segment information in the Plücker coordinate system is transformed into the camera coordinate system.

[0034] The spatial straight line in the camera coordinate system is re-projected onto the image plane, and the re-projection error of the straight line is represented by the perpendicular distance from the coordinates of the line segment endpoints to the straight line on which the line segment is located.

[0035] The second aspect of the present application provides a binocular vision positioning system in a weak light environment based on deep learning enhancement.

[0036] The binocular vision positioning system in a weak light environment based on deep learning enhancement includes:

[0037] An image acquisition module configured to acquire binocular vision images.

[0038] An edge feature enhancement module configured to perform edge feature enhancement on the binocular vision images using an edge detection model.

[0039] A line feature extraction module configured to perform line feature extraction on the binocular vision images after edge feature enhancement using a line detection algorithm, and to remove false line segments caused by point light source scattering using a light erasing logic.

[0040] A point feature extraction and point-line feature matching module configured to extract point features of the binocular vision images using an image point feature extraction model, to perform feature matching based on the acquired point features using a SuperGlue model, and to determine the positional relationship between the point features and the line features and to obtain the relationship of line feature matching using the relationship of point feature matching.

[0041] The backend optimization module is configured to: based on a key frame selection strategy, screen frames with position, angle change and low feature matching number for backend optimization, and output a final positioning trajectory; the backend optimization comprises optimizing inter-frame poses by constructing a re-projection residual error model of point and line features.

[0042] The third aspect of the application provides a computer readable storage medium, which stores a program, and the program is executed by a processor to realize the steps in the binocular vision positioning method in a weak light environment based on deep learning enhancement according to the first aspect of the application.

[0043] The fourth aspect of the application provides an electronic device, which comprises a memory, a processor and a program stored in the memory and executable on the processor, and the processor executes the program to realize the steps in the binocular vision positioning method in a weak light environment based on deep learning enhancement according to the first aspect of the application.

[0044] The above one or more technical solutions have the following beneficial effects:

[0045] (1) The application can effectively improve the feature extraction stability in a weak light environment. The RCF (Richer Convolutional Features) model is used to enhance the edges of the image, effectively improving the extraction effect of the line features in the weak light condition, and effectively reducing the feature loss problem caused by noise interference in the traditional edge detection method. The combination of the SuperPoint and SuperGlue models makes the point feature extraction and matching still maintain high precision and robustness in low light conditions, and reduces the false matching rate.

[0046] (2) The application automatically identifies and eliminates false line features generated by point light scattering through light scattering suppression logic, avoids the influence of such interference on pose solution, improves the reliability of line features in a weak light environment, and effectively suppresses light scattering interference.

[0047] (3) By using the Plucker coordinate to represent the space line, the re-projection error calculation of the line feature is optimized, and the accuracy of the pose solution is further improved. By dynamically selecting key frames, the redundant calculation is reduced, the real-time performance of the system is improved while ensuring the positioning accuracy, and the application scenarios with limited computing resources (such as mobile robots, unmanned aerial vehicles, etc.) are suitable.

[0048] The advantages of the additional aspects of the application will be partially given in the following description, partially will become obvious from the following description, or will be learned by the practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0049] The accompanying drawings, which form a part of this specification, are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification. The embodiments of the application, and their

[0050] Figure 1 The method flow chart of the first embodiment.

[0051] Figure 2 The model architecture diagram of the edge detection model in the first embodiment.

[0052] Figure 3 The line feature extraction effect diagram after edge enhancement in the first embodiment.

[0053] Figure 4 The line feature extraction effect diagram without edge enhancement in the first embodiment.

[0054] Figure 5 The effect diagram after point light source light scattering removal in the first embodiment.

[0055] Figure 6 The effect diagram before point light source light scattering removal in the first embodiment.

[0056] Figure 7 The point feature and line feature matching principle diagram in the first embodiment.

[0057] Figure 8 The comparison result diagram between the method of the present application and other methods in the first embodiment.

[0058] Figure 9 The system structure diagram of the second embodiment. DETAILED DESCRIPTION

[0059] It should be noted that the following detailed description is merely exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0060] It is to be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments according to the present application.

[0061] In the case of no conflict, the embodiments in the application and the features in the embodiments can be combined with each other.

[0062] Embodiment one

[0063] The embodiment discloses a binocular vision positioning method in a weak light environment based on deep learning enhancement.

[0064] As Figure 1As shown, the binocular vision positioning method in a weak light environment based on deep learning enhancement comprises:

[0065] In step S1, binocular vision images are acquired; the binocular vision images comprise left images and right images, and the left and right view images are synchronously acquired by a calibrated binocular camera; and the acquired images are timestamped and geometrically corrected.

[0066] In step S2, an edge detection model is used to enhance the edge features of the binocular vision images.

[0067] The edge detection model (RCF) is a convolutional neural network model for image edge inference. Traditional image edge detection methods mainly focus on image color gradient, and special algorithms are designed to traverse the image to obtain edge features. However, when performing visual SLAM line feature extraction, the traditional edge detection method is not stable enough in extracting line features between frames in a dark imaging environment. The edge detection model uses a full convolutional network to combine the convolutional features of other intermediate layers to achieve the function of fusing more image details.

[0068] Further, the structure and function of the edge detection model include the following contents:

[0069] In combination with Figure 2 , the edge detection model is improved based on VGG16, and all fully connected layers and the fifth layer pooling layer are deleted; each convolutional layer is connected with a convolutional layer with a kernel of ; followed by an element-wise operation (eltwise) layer; then an up-sampling is performed on the layer feature map using a deconvolutional layer; the results after up-sampling are connected through a Sigmoid cross-entropy loss; all up-sampling results of the convolutional layers are fused, and then a convolutional layer with a kernel of is connected to calculate the final loss.

[0070] The kernels of the main convolutional layers of the edge detection model are , and the dimensions of the input images are arbitrary. After multiple convolution, padding and merging, a picture with the same dimension as the input image is obtained.

[0071] In step S3, a line detection algorithm is used to extract line features from the binocular vision images after edge feature enhancement, and a light erasing logic is used to remove false line segments caused by point light source scattering; the effect is shown in Figures 3-6 . Among them, Figure 3 , Figure 4 show the line feature extraction effect before and after edge enhancement. The red line segments in the figure represent the extracted line features. Figure 5 , Figure 6 show the effect comparison before and after the removal of point light source light scattering. The red line segments in the figure represent the extracted line features, and the green serial numbers represent the serial numbers of the extracted line features.

[0072] Extracting line features of binocular vision image enhanced by straight line detection algorithm (LSD) and geometric center coordinates of light source in image ;

[0073] Extract the pixel value corresponding to the end point of all lines, and stretch the contrast of the line segment end point, and set the astigmatism threshold ,

[0074]

[0075] Among them, , , is the pixel value of the line segment end point after contrast stretching; , , respectively represent the original image channel value of the line segment end point; , , is the image stretching coefficient; , , is the pixel value offset.

[0076] The pixel value of the end point after contrast stretching is calculated by gray value, and the line segment whose starting point gray value is greater than the astigmatism threshold is further excluded, and the vertical distance from the center coordinate to each line is calculated. Whether it is caused by astigmatism is judged by distance threshold.

[0077] The pixel value of the end point after contrast stretching is calculated by gray value.

[0078]

[0079] Among them, is the pixel gray value corresponding to the line feature end point coordinate; , , is the weight coefficient of the image channel.

[0080] The line segment whose starting point gray value is greater than the astigmatism threshold is further excluded to obtain ( ); and calculate the vertical distance from the center coordinate to each line. If it is less than a certain threshold, it can be considered that the line is caused by astigmatism.

[0081]

[0082] Among them, represents the straight line distance from the ith light center to the nth line segment; 、 represents the pixel coordinate of the i-th optical center; 、 、 、 represents the starting pixel coordinate of the n-th line segment. When is less than the distance threshold , it can be considered that the line segment is generated due to light scattering. The parameters used above can be appropriately adjusted before program running to achieve different degrees of suppression effect. Through the above processing, the influence of false line features generated by astigmatism on pose solving can be greatly suppressed.

[0083] In step S4, the point features of the binocular vision image are extracted using an image point feature extraction model. Based on the obtained point features, a SuperGlue model is used for feature matching. The positional relationship between the point features and the line features is judged, and the relationship of the line feature matching is obtained using the relationship of the point feature matching.

[0084] In this embodiment, the image point feature extraction model uses a SuperPoint model, and the image matching model uses a SuperGlue model.

[0085] SuperPoint is a kind of self-supervised neural network for extracting picture point features and descriptors. In this embodiment, the feature points of each frame are directly inferred by the SuperPoint model, and the number of feature points of each frame is preset when the program starts.

[0086] In step S41, in the process of extracting the point features of the binocular vision image using the SuperPoint model, the binocular vision image is first compressed by an encoder. The encoder compresses the binocular vision image through a VGG-style convolutional network for three times of convolution, pooling and nonlinear activation, realizes the compression of the binocular vision image, and the size of the compressed image becomes 1 / 8*1 / 8 of the original image. The encoded tensor will simultaneously detect the position of the feature points and output the descriptors.

[0087] The detection output dimension of the point features is , wherein the 64 channels correspond to non-overlapping pixel grids in the original image. The extra one-dimensional dimension is called the dustbin dimension, which is used to suppress the large value generated by softmax when there is no feature point in the region. After softmax, reshaping can obtain a tensor with the same size as the binocular vision image.

[0088] For the point feature information output by SuperPoint, high-confidence point feature information is selected, and the feature points are evenly distributed on the binocular vision image through grid allocation.

[0089] Step S42, further, for the descriptor obtained by the SuperPoint model, an attention mechanism and a graph neural network are used to optimize feature point matching, and a dustbin mechanism is used to filter false matching pairs.

[0090] SuperGlue is a neural network that combines query feature point pairs and excludes mismatched feature points. In combination with the output of the SuperPoint model, better extraction and matching of feature points in different frames can be obtained.

[0091] Specifically, based on the descriptors obtained by the image point feature extraction model, an attention mechanism and a graph neural network are used to optimize feature point matching, and a dustbin mechanism is used to filter false matching pairs.

[0092] For pictures A and B, i.e. the left and right images of binocular vision, each has a set of feature points p and corresponding descriptors d, define (p, d) as a set of local features, where the descriptor d is generated by the SuperPoint model, then the local feature set of the binocular vision image can be expressed as The local feature set of the binocular vision image can be expressed as .

[0093] An attention graph neural network is used to simulate the behavior of human observation of feature point matching, and the feature point position and description information are bound. The combined initial representation of each feature point is ,

[0094]

[0095] where MLP represents a multi-layer perceptron, and is the position and descriptor of the i-th feature point, the low-dimensional feature point is upgraded, and the point description and point position are coupled; A graph neural network is used, and each feature point in the image is represented as a node on the graph. There are two kinds of edges in the graph, the intra-image edge is represented as

[0096] (connecting the feature points of the same graph), and the inter-image edge is represented as (connecting the feature points of the same graph). Define is the i-th element of image A at the i-th layer, and the information is the aggregation result of all feature points, define is the connection operator, then the residual error information of all features of image A after updating is:

[0097] ​​

[0098] Where, is the i-th element in image A at The residual information after the feature transfer layer is updated;

[0099] Perform aggregation and count messages using attention mechanism The inner edge of the image is based on the inner edge of the image, and the outer edge of the image is based on the outer edge of the image. Similar to database retrieval, that is, query , according to the attributes of some elements (key ) retrieve the value of certain elements .

[0100]

[0101] Among them, the attention weight , is the representation of all feature points, and .key , query Sum is calculated as the linear projection of the deep features of the graph neural network. Consider the query feature point i in image Q, and all source feature points are on image S, where , then the key , query Sum Expressed as:

[0102]

[0103] Each layer Each has its own projection parameters, which can be learned and shared for all feature points in the two images.

[0104] After L times of calculation of the inner and outer edges of the image, the output of the attention GNN is obtained. The final matching descriptor for image A is expressed as:

[0105]

[0106] Next, SuperGlue constructs the optimal matching layer, which generates a distribution matrix P, which can be calculated by a score matrix This is achieved by maximizing the score The P matrix can be obtained to express the pairwise scores as the similarity of matching descriptors:

[0107]

[0108] in, It is the inner product operation.

[0109] To let the network suppress certain key points, the model uses dustbin to augment each set of feature points so as to explicitly assign unmatched feature points to it. This technique is common in graph matching, and SuperPoint also uses dustbin to account for image cells that may not be detected, and SuperGlue sets the last row and last column of the score matrix to dustbin to filter out erroneous matching feature pairs, as shown in the following formula:

[0110]

[0111] After obtaining the matching relationship of the point features, for the point and line features on the same image, it is determined that which point features are on the line features or the position of the point features is less than the set matching threshold distance from the position of the line features, and then the point features are defined to belong to the line features. According to the matching relationship of the point features, the matching relationship of the line features is further obtained.

[0112] Specifically, based on the point features and the matching relationship of the point features, each line feature is traversed to calculate whether the point features in the left image are on the line feature, and the following requirements need to be met:

[0113] a. The projection of the point feature on the coordinate axis should fall within the projection of the end points of the line feature on the coordinate axis;

[0114] b. The distance of the point to the straight line on which the line segment is located cannot be greater than the set distance threshold.

[0115] The logic diagram is shown in Figure 7 , and the red points in the figure do not meet the point: from this, the attribution relationship of the point features and the line features on the left and right images can be obtained, and then the matching relationship of the line features can be obtained according to the matching relationship of the point features.

[0116] Step S5, based on the key frame selection strategy, frames with position, angle change and low feature matching number are screened for backend optimization, and the final positioning trajectory is output; the backend optimization includes optimizing the inter-frame pose by constructing a point and line feature reprojection residual model. Wherein, the process of optimizing the pose by constructing the point reprojection residual model includes:

[0117] Restoring the depth information of the feature points by binocular stereo vision;

[0118] The 3D points in the world coordinate system are converted into the camera coordinate system and normalized to obtain the pixel coordinates of the point features after re-projection, which are represented as follows:

[0119]

[0120] wherein, is the re-projection error of the feature points.​ is the observed pixel coordinate of the feature point in the k-th key frame; is the position of the feature point in the world coordinate system; and are the transformation matrix from the camera coordinate system to the pixel coordinate system and the transformation matrix from the world coordinate system to the camera coordinate system, respectively.

[0121] Further, in the process of constructing the re-projection residual model of the line feature to optimize the inter-frame pose, the Plücker coordinate system is introduced to construct the re-projection residual model of the line feature to optimize the pose, and the process includes:

[0122] After the extraction of the image line feature, 2D line segment information is obtained. Since it is a binocular SLAM system, the spatial 3D line segment information is obtained by calculating the line segment endpoint depth, that is, the 3D line segment is represented by the 3D points in the two camera coordinate systems. Using the spatial rectangular coordinate system to describe the straight line in space will have redundant degrees of freedom, so the Plücker coordinate system is used for expression, that is, the spatial straight line equation is:

[0123]

[0124] wherein, is the representation of the straight line in the Plücker coordinate system, is the direction vector of the straight line, is the cross product of the vectors composed of the two 3D endpoints of the line segment and the origin.

[0125] The mathematical relationship of transforming the straight line from the world coordinate system to the camera coordinate system is:

[0126]

[0127] wherein, is the expression form of the Plücker coordinate of the straight line in the camera coordinate system and the world coordinate system, is the rotation matrix and the transformation matrix from the world coordinate system to the camera coordinate system, is the anti-symmetric matrix representation of the three-dimensional vector.

[0128] Space straight line in the camera coordinate system The transformation formula of re-projecting to the image plane is:

[0129]

[0130] wherein, is the representation of the spatial straight line in the two-dimensional plane, is the coefficient thereof, is the camera intrinsic parameter, is the projection matrix from the camera coordinate system to the image coordinate system.​

[0131] The straight line represented by the Plucker coordinates can be re-projected from the world coordinate system to the image plane by the above transformation, and the re-projection error of the line feature is represented by the perpendicular distance from the line segment endpoint coordinates to the straight line on which the line segment lies, that is:

[0132]

[0133] wherein, is the re-projection error of the straight line, is the calculation method of the distance from the point to the straight line, are the two endpoint coordinates of the planar line segment.

[0134] Further, by selecting the OIVIO dataset, the OIVIO dataset is an open source dataset of visual slam, and the collection environment is a low-light underground tunnel, pipeline and the like, and the trajectory true value is provided as a comparison, as shown in Figure 8 Further, the results of the calculation under the interception method and the trajectory true value are compared, and the RMSE results of the absolute pose error are shown in Table 1:

[0135] Table 1 RMSE results of absolute pose error

[0136]

[0137] From Table 1, it can be seen that the RMSE results of the absolute pose error of the method provided by the application are obviously better than the calculation results of other methods, and have better performance in a low-light and low-texture environment. Further, it is further indicated that in a low-light environment, simultaneously using point and line features can make the front end of visual SLAM more robust.

[0138] Embodiment two

[0139] The embodiment discloses a binocular vision positioning system in a weak light environment based on deep learning enhancement;

[0140] As shown in Figure 9 , the binocular vision positioning system in a weak light environment based on deep learning enhancement comprises:

[0141] An image acquisition module configured to acquire binocular vision images;

[0142] An edge feature enhancement module configured to perform edge feature enhancement on the binocular vision images by using an edge detection model;

[0143] A line feature extraction module configured to perform line feature extraction on the binocular vision images after edge feature enhancement by using a straight line detection algorithm, and to remove false line segments caused by point light source scattering by using a light erasing logic;

[0144] The point feature extraction and matching module is configured to: extract point features of binocular vision images using an image point feature extraction model; perform feature matching using a SuperGlue model based on the obtained point features; determine the positional relationship between the point features and the line features, and use the relationship of point feature matching to obtain the relationship of line feature matching.

[0145] The backend optimization module is configured to: based on a key frame selection strategy, screen frames with position, angle changes and low feature matching numbers for backend optimization, and output a final positioning trajectory; the backend optimization includes optimizing inter-frame poses by constructing a re-projection residual model of point and line features.

[0146] Embodiment three

[0147] The purpose of the present embodiment is to provide a computer-readable storage medium.

[0148] A computer-readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the steps of the binocular vision positioning method in a weak light environment based on deep learning enhancement according to embodiment 1.

[0149] Embodiment four

[0150] The purpose of the present embodiment is to provide an electronic device.

[0151] An electronic device, comprising a memory, a processor, and a program stored on the memory and executable on the processor, wherein the processor implements the steps of the binocular vision positioning method in a weak light environment based on deep learning enhancement according to embodiment 1 when executing the program.

[0152] The steps and methods involved in the above embodiments two, three and four correspond to embodiment one, and the specific embodiments can be referred to the relevant description part of embodiment one. The term "computer-readable storage medium" should be understood to include a single medium or multiple media of one or more instruction sets; it should also be understood to include any medium capable of storing, encoding or carrying instruction sets for execution by a processor and causing the processor to perform any of the methods in the present application.

[0153] Those skilled in the art should understand that each module or step of the present application described above can be realized by a general computer device, alternatively, they can be realized by program codes executable by a computing device, so that they can be stored in a storage device for execution by a computing device, or they can be made into individual integrated circuit modules, or a plurality of modules or steps among them can be made into a single integrated circuit module. The present application is not limited to any specific combination of hardware and software.

[0154] The above describes the specific embodiments of the present application in combination with the drawings, but is not a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications or variations made by those skilled in the art on the basis of the technical solutions of the present application without creative labor are still within the protection scope of the present application.

Claims

1. A binocular vision positioning method in a weak light environment based on deep learning enhancement, characterized in that, The method comprises the following steps: acquiring binocular vision images; using an edge detection model to enhance the edge features of the binocular vision images; the edge detection model is based on an improved VGG16 network structure, and full connection layers and a fifth layer of pooling layers are deleted, and a multi-scale feature fusion is used to enhance the edge detection capability; using a straight line detection algorithm to extract line features from the binocular vision images after the edge feature enhancement, and using a light ray erasing logic to remove false line segments caused by point light source scattering, comprising: extracting the line features of the binocular vision images after the edge feature enhancement and the geometric center coordinates of the light sources in the images; extracting the pixel values corresponding to the endpoints of all lines, and performing contrast stretching on the line segment endpoints, and setting a glare threshold; calculating the gray value of the pixel values of the endpoints after the contrast stretching, further excluding the line segments whose starting point gray values are all greater than the glare threshold, and calculating the perpendicular distance from the light source center coordinates to each line, and judging whether it is caused by glare through a distance threshold; using an image point feature extraction model to extract point features of the binocular vision images; based on the obtained point features, using a SuperGlue model to perform feature matching; judging the positional relationship between the point features and the line features, using the point feature matching relationship to obtain the line feature matching relationship, comprising: based on the descriptors obtained by the image point feature extraction model, using an attention mechanism and a graph neural network to optimize the feature point matching, and filtering the wrong matching pairs through a dustbin mechanism; after obtaining the matching relationship of the point features, for the point and line features on the same image, judging which point features are on the line features or the position distance between the point features and the line features is less than a set matching threshold, then defining that the point features belong to the line features, and according to the matching relationship of the point features, further obtaining the matching relationship of the line features; based on a key frame selection strategy, screening frames with position, angle change and low feature matching number for backend optimization, and outputting a final positioning trajectory; the backend optimization comprises optimizing the inter-frame pose through a re-projection residual model of the point and line features.

2. The method of claim 1, wherein the method is based on deep learning enhancement. using an image point feature extraction model to extract point features of the binocular vision images, comprising: the image point feature extraction model compresses the binocular vision images through an encoder, detects the feature point positions of the compressed images, and outputs descriptors; the results of the feature point position detection are subjected to a softmax function and a reshape operation to obtain a tensor with the same size as the binocular vision images; selecting high-confidence point feature information, and distributing the point features uniformly on the binocular vision images through grid allocation.

3. The method of claim 1, wherein the method is based on deep learning enhancement. the process of optimizing the image pose through the re-projection residual model of the point features comprises: after extracting the point features and matching, the depth information of the image points is recovered through binocular stereo vision; transforming the 3D points in the world coordinate system to the camera coordinate system; the pixel error formed by the image positions after the 3D point transformation and the extracted feature points constitutes the re-projection error of the point features.

4. The method of claim 1, wherein the method is based on deep learning enhancement. the process of optimizing the inter-frame pose through the re-projection residual model of the line features comprises: calculating the line segment endpoint depth of the extracted line features to obtain the information of the spatial 3D line segments; The space 3D line segment is expressed by using Plucker coordinates to obtain line segment information in the Plucker coordinate system; The line segment information in the Plucker coordinate system is transformed into a camera coordinate system; The space straight line in the camera coordinate system is re-projected onto an image plane, and the re-projection error of the straight line is represented by the vertical distance from the line segment endpoint coordinates to the straight line.

5. The binocular vision positioning system based on deep learning enhancement in weak light environment, characterized in that: It comprises: An image acquisition module configured to acquire binocular vision images; An edge feature enhancement module configured to enhance edge features of the binocular vision images by using an edge detection model; The edge detection model is based on an improved VGG16 network structure, and the full connection layer and the fifth layer pooling layer are deleted, and the edge detection capability is enhanced through multi-scale feature fusion; A line feature extraction module configured to extract line features of the binocular vision images after edge feature enhancement by using a straight line detection algorithm, and to remove false line segments caused by point light source scattering by using light erasing logic, comprising: extracting line features and geometric center coordinates of light sources in the binocular vision images after edge feature enhancement; extracting pixel values corresponding to endpoints of all lines, and performing contrast stretching on the line segment endpoints, and setting a glare threshold; calculating the gray value of the contrast stretched endpoint pixel value, further excluding the line segment whose starting point gray value is greater than the glare threshold, and calculating the vertical distance from the light source center coordinates to each line, and determining whether it is caused by glare through the distance threshold; A point feature extraction and point-line feature matching module configured to extract point features of the binocular vision images by using an image point feature extraction model; based on the obtained point features, performing feature matching by using a SuperGlue model; determining the positional relationship between the point features and the line features, and using the point feature matching relationship to obtain the line feature matching relationship, comprising: based on the descriptors obtained by the image point feature extraction model, using attention mechanism and graph neural network to optimize feature point matching, and filtering error matching pairs through dustbin mechanism; After obtaining the matching relationship of the point features, for the point and line features on the same image, it is determined which point features are on the line features or the position distance between the point features and the line features is less than the set matching threshold, then the point features are defined as belonging to the line features, and the matching relationship of the line features is obtained according to the matching relationship of the point features; A backend optimization module configured to filter frames with position, angle change and low feature matching number for backend optimization based on a key frame selection strategy, and output a final positioning trajectory; the backend optimization comprises optimizing inter-frame poses by constructing a re-projection residual model of point and line features.

6. A computer-readable storage medium having stored thereon a program, characterized in that, The program is executed by the processor to realize the steps in the weak light environment based on deep learning enhanced binocular vision positioning method in any one of claims 1-4.

7. An electronic device comprising a memory, a processor, and a program stored on the memory and executable on the processor, characterized in that, The processor executes the program to realize the steps in the weak light environment based on deep learning enhanced binocular vision positioning method in any one of claims 1-4.

Images