A visual positioning method, device and medium for a tunnel scene

Abstract

The application provides a tunnel scene-oriented visual positioning method, device and medium, and belongs to the field of visual positioning. The method comprises the following steps: acquiring image data of a tunnel area shot by an imaging component on a test vehicle; the image data comprises a color image and a depth image; acquiring motion data collected by an inertial sensor on the test vehicle; based on an oFAST algorithm, a FREAK algorithm, a non-maximum suppression algorithm and a random sample consensus algorithm, optimizing feature points extracted from the color image to obtain a feature point set uniformly distributed; based on a Viterbi algorithm and a second-order hidden Markov model, filtering out a reference image sequence according to the color image, the depth image and the motion data; based on an improved target detection model, extracting tunnel scene features in the tunnel according to the color image; and obtaining a current pose of the test vehicle according to the tunnel scene features, the reference image sequence and the depth image. The application improves the positioning accuracy in the tunnel scene.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to a visual positioning method, device, and medium for tunnel scenarios. Background Technology

[0002] With the development of rail transit, many jobs in dark environments are difficult and dangerous, leading many companies to adopt automated methods. However, the biggest problem with automation is positioning accuracy. Failure to solve this problem will significantly impact project progress.

[0003] Existing GPS or BeiDou positioning systems often experience signal loss in many tunnel scenarios because these satellite positioning methods are typically used for wider outdoor positioning where high precision is not required. Visual positioning in tunnel scenarios is of great significance for projects such as tunnel construction and bridge construction, and is very useful for many hazardous tasks, such as installing cables, supports, shelves, drilling holes, or replacing warning signs. Moreover, compared to LiDAR positioning, visual positioning is less expensive, requiring only RGB cameras, depth cameras, and inertial measurement units, while still providing accurate and robust location information.

[0004] Currently, besides satellite positioning technologies like GPS, mainstream positioning technologies all utilize sensors for localization, including UWB positioning, LiDAR positioning, camera positioning, and WiFi positioning. LiDAR and camera positioning technologies are typically included in SLAM, which involves simultaneous localization and map building. LiDAR-based SLAM is called LiDAR SLAM, while camera-based SLAM is called Visual SLAM, and there are also multi-sensor fusion SLAM technologies. The key to SLAM is the construction of feature maps. In Visual SLAM, traditional algorithms can be used to represent scene information, or deep learning methods can be used to train models and extract semantic information to describe the scene. Traditional feature extraction algorithms include SIFT, SURF, and ORB, and researchers have also improved upon them. Deep learning methods are more recent developments, with common models including Swim-Transformer, SuperPoint, and SuperGlue. Traditional feature extraction algorithms struggle to accurately extract features in scenes with poor lighting conditions, high repetition, and low texture, such as tunnels, leading to inaccurate localization within tunnels. Summary of the Invention

[0005] In view of this, it is necessary to provide a visual positioning method, device and medium for tunnel scenes to solve the technical problem in the prior art that it is difficult to accurately extract features in scenes such as tunnels with poor lighting conditions, high repetition and low texture, which leads to inaccurate positioning in tunnels.

[0006] To address the aforementioned technical problems, in a first aspect, the present invention provides a visual positioning method for tunnel scenarios, comprising:

[0007] Acquire image data of the tunnel area captured by the imaging component on the test vehicle; the image data includes color images and depth images;

[0008] Acquire motion data collected by the inertial sensors on the test vehicle;

[0009] Based on the oFAST algorithm, FREAK algorithm, nonmaximum suppression algorithm and random sampling consensus algorithm, the feature points extracted from the color image are optimized to obtain a uniformly distributed feature point set.

[0010] Based on the Viterbi algorithm and the second-order hidden Markov model, a reference image sequence is selected according to the feature point set, the depth image, and the motion data;

[0011] Based on the improved target detection model, tunnel scene features in the tunnel are extracted from the color image;

[0012] The current pose of the test vehicle is obtained based on the tunnel scene features, the reference image sequence, and the depth image.

[0013] In one possible implementation, the optimization processing of feature points extracted from the color image based on the oFAST algorithm, FREAK algorithm, nonmaximum suppression algorithm, and random sample consensus algorithm to obtain a uniformly distributed feature point set includes:

[0014] The color image is processed to obtain a grayscale image;

[0015] The grayscale image is subjected to gamma correction to obtain the processed image;

[0016] ;or,

[0017] The grayscale image is normalized to obtain the processed image;

[0018] ;

[0019] in, To output grayscale values, The grayscale value of the input grayscale image. For compensation coefficient, Gamma coefficient, It is a positive number;

[0020] Based on the multi-scale pyramid algorithm, enhanced images of multiple scales are generated from the processed image; the aspect ratio of the enhanced images at two adjacent scales is the same.

[0021] Based on the oFAST algorithm, feature points are extracted from the enhanced image at multiple scales. The orientation angle of the feature point is calculated according to the first coordinates corresponding to the feature point and the second coordinates corresponding to the centroid. The centroid is a point within a neighborhood circular region with the feature point as the center and the neighborhood radius as the radius.

[0022] , , ;

[0023] in, The neighborhood radius, In the enhanced image Moments within the range, , For order, , pixel coordinates of Power and Power of 1 For the first The grayscale value of each pixel Let the coordinates of the centroid of the moment be... It is the sum of the gray values ​​of all pixels in the neighborhood. For the grayscale value of a pixel in the neighborhood Weighted sum of distributions in different directions For the grayscale value of a pixel in the neighborhood Weighted sum of distributions in different directions;

[0024] Generate a binary descriptor for each feature point based on the FREAK algorithm;

[0025] ;

[0026] ;

[0027] in, For binary descriptors, For the index variable in the summation process, The length of the binary code. For a sampling point pair, For sampling point pairs The pixel value of the previous sampling point, This is the pixel value of the next sampling point;

[0028] Based on the nonmaximum suppression algorithm and the random sampling consensus algorithm, the feature points corresponding to the binary descriptor are optimized to obtain the feature point set.

[0029] In one possible implementation, the step of optimizing the feature points corresponding to the binary descriptor based on the nonmaximum suppression algorithm and the random sample consensus algorithm to obtain the feature point set includes:

[0030] Centered on the feature point, select its surrounding neighborhood. The average gray value of the feature point is calculated from each pixel.

[0031] ;

[0032] in, For the first The average gray value of each feature point;

[0033] Based on the non-maximum suppression algorithm, non-maximum feature points are selected from a local region centered on the feature point and with a local region radius as the radius; the non-maximum feature point is the feature point in the local region whose average gray value is not the largest.

[0034] The target feature point is obtained by performing preliminary optimization on the non-maximum feature points based on the suppression radius;

[0035] ;

[0036] in, The suppression radius is negatively correlated with the distinctiveness of the suppression region. Non-maximum feature points The corresponding Gaussian difference value, Non-maximum feature points The corresponding Gaussian difference value, The symbol for Euclidean distance calculation is... The constraint symbol indicates that conditions must be satisfied. Non-maximum feature points ;

[0037] Based on the random sampling consensus algorithm, an inlier set is obtained according to the target feature points; the inlier set includes multiple pairs of inlier samples.

[0038] A feature model is created based on the inlier samples. The remaining feature points are evaluated based on the feature model. The remaining feature points evaluated as inliers are added to the inlier set. The feature model is updated based on the inlier set. The update is iterated until the proportion of inliers reaches the target, and then the feature set is obtained.

[0039] In one possible implementation, the step of selecting a reference image sequence based on the Viterbi algorithm and a second-order hidden Markov model, according to the feature point set, the depth image, and the motion data, includes:

[0040] Create a vehicle kinematic model for the test vehicle; the speed of the test vehicle is constant.

[0041] ;

[0042] in, For the test vehicle at time Location, For the test vehicle at time Location, For the test vehicle at time The position, the interval between two adjacent moments is less than a preset value;

[0043] If the first acquisition frame rate of the imaging component is inconsistent with the second acquisition frame rate of the inertial sensor, the motion data is interpolated to ensure that the feature point set, the depth image and the motion data are time-aligned.

[0044] After aligning the feature point set, the depth image, and the motion data in pose, a subset of candidate nodes is obtained based on the real-time pose of the test vehicle.

[0045] Based on the second-order hidden Markov model and the Viterbi algorithm, the reference image sequence corresponding to the undetermined observation sequence is obtained according to the candidate node subset.

[0046] In one possible implementation, obtaining a subset of candidate nodes based on the real-time pose of the test vehicle includes:

[0047] Calculate the first distance value between the real-time pose and each map node in the map node set;

[0048] The nearest map node is selected from the set of map nodes, and the first distance value between the nearest map node and the real-time pose is the smallest.

[0049] Candidate map nodes are selected to obtain the candidate node subset. The second distance value between the candidate map node and the nearest map node is less than a preset threshold. The candidate node subset includes the nearest map node and all the candidate map nodes.

[0050] In one possible implementation, obtaining the reference image sequence corresponding to the undetermined observation sequence based on the candidate node subset according to the second-order hidden Markov model and the Viterbi algorithm includes:

[0051] Based on the Viterbi algorithm, the joint probability is obtained by recursively calculating the hidden state sequence and the undetermined observation sequence; the undetermined observation sequence is established based on the image data at the current time.

[0052] ;

[0053] The maximum joint probability is calculated based on the posterior probability and the joint probability.

[0054] ;

[0055] in, For the maximum joint probability, For a moment The hidden state below, From time 1 to time 2 The observation sequence, Proportional to the sign, Let be the posterior probability, representing the probability given from time 1 to time 2. observation sequence Under the condition at time Hidden state The probability of occurrence For a moment The joint probability, representing time... Hidden state From time 1 to time 2 observation sequence The probability of them happening simultaneously;

[0056] The initial state probability is calculated based on the position of the test vehicle;

[0057] , ;

[0058] in, This represents the map node index corresponding to the vehicle's actual starting position. It is a general index when traversing map nodes. This indicates that the test vehicle originated from a map node. The probability of departure. The initial state probability;

[0059] The state transition probability is calculated based on the hidden state sequence and the position of the test vehicle on the map node;

[0060] ;

[0061] ;

[0062] in, , and All are map nodes. For the test vehicle at map nodes Location, The first covariance;

[0063] The emission probability is calculated based on the image features and the number of feature points of the current image to be matched by the test vehicle at time t.

[0064] ;

[0065] in, The hidden state at time t Compared with observed values The number of feature points matched between, mean This is the number of ORB feature points obtained from the current image to be matched;

[0066] Based on the chain rule and the properties of the second-order hidden Markov model, the forward probability is calculated according to the emission probability and the state transition probability.

[0067] ;

[0068] ;

[0069] in, For a moment The joint probability, For a moment The probability of launch, For a moment The state transition probability, For a moment The forward probability, For a moment The forward probability;

[0070] Based on the posterior probability, the maximum joint probability, and the time... The forward probability is used to calculate the reference image sequence.

[0071] In one possible implementation, the step of extracting tunnel scene features from the color image based on the improved target detection model includes:

[0072] An object detection model is obtained by adding an attention mechanism and a spatial feature fusion mechanism to the Yolov5s network model.

[0073] The target detection model is trained on the training set until the model parameters are determined to meet the standards based on the validation set, thus obtaining the improved target detection module; both the training set and the validation set include image samples within the tunnel area, and the image samples are labeled with key features;

[0074] Based on the YOLO algorithm and the improved target detection module, the tunnel scene features are extracted from the processed image obtained from the color image.

[0075] In one possible implementation, obtaining the current pose of the test vehicle based on the tunnel scene features, the reference image sequence, and the depth image includes:

[0076] Select target feature images from the reference image sequence that have the same features as the tunnel scene;

[0077] Filter the target depth image that is captured at the same time as the target feature map, and obtain the corresponding depth information based on the target depth image;

[0078] A similar image set is created based on the reference image sequence; the similar image set includes multiple similar images, each of which includes a first target feature image and a second target feature image, wherein the first target feature image is the target feature image in the reference image sequence in which the tunnel scene features are complete, and the second target feature image is the target feature image in the reference image sequence in which the tunnel scene features are incomplete;

[0079] The two-dimensional coordinates of the test vehicle are calculated based on the processed image and the similar image acquired at the current moment, and the current pose is calculated based on the two-dimensional coordinates and the depth information.

[0080] In a second aspect, the present invention also provides an electronic device, including a memory and a processor, wherein,

[0081] The memory is used to store programs;

[0082] The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps in the visual positioning method for tunnel scenarios described in any of the above implementations.

[0083] Thirdly, the present invention also provides a computer-readable storage medium for storing a computer-readable program or instructions, which, when executed by a processor, can implement the steps of the visual positioning method for tunnel scenarios described in any of the above implementations.

[0084] The beneficial effects of this invention are as follows: The visual positioning method, device, and medium for tunnel scenes provided by this invention first acquire color and depth images by using an imaging component installed on a test vehicle. An inertial sensor installed on the test vehicle collects motion data. Based on the oFAST algorithm, feature points in the color image are quickly detected, exhibiting rotation invariance and suitable for extracting key points in complex scenes such as tunnels. Based on the FREAK algorithm, feature points are described to generate efficient binary descriptors, improving the speed and accuracy of feature matching. Based on the non-maximum suppression algorithm, redundant feature points are removed, making the feature point distribution more uniform, improving matching efficiency and robustness. Based on the random sampling consensus algorithm, mismatched feature points are removed through random sampling and consensus checks, further optimizing to obtain a uniformly distributed feature point set. Furthermore, based on the Viterbi algorithm and a second-order hidden Markov model, the feature point set, depth image, and motion data are used to select the reference image sequence most similar to the image to be matched, effectively handling the problems of repetitive and low-texture textures in tunnel scenes, improving matching accuracy and robustness. Furthermore, based on the improved target detection model combined with color and depth images, scene features (such as valves and doorways) in the tunnel are extracted, and the current pose of the test vehicle is calculated using reference image sequences and depth information. This can achieve centimeter-level positioning accuracy and is suitable for precise positioning in complex environments with low texture and repetitive texture, such as tunnels. Attached Figure Description

[0085] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0086] Figure 1 This is a schematic flowchart of an embodiment of the visual positioning method for tunnel scenarios provided by the present invention;

[0087] Figure 2 This is a schematic diagram of the vehicle model used for testing the vehicle of this invention;

[0088] Figure 3 This is a flowchart of an embodiment of the visual positioning method for tunnel scenarios provided by the present invention;

[0089] Figure 4 This is a schematic diagram of sequence matching provided by the present invention;

[0090] Figure 5 The ASFF schematic diagram provided for this invention;

[0091] Figure 6The feature detection effect diagram provided by the present invention;

[0092] Figure 7 This is a schematic diagram of the matching sequence filtering provided by the present invention;

[0093] Figure 8 A schematic diagram of an embodiment of the electronic device provided by the present invention. Detailed Implementation

[0094] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0095] In the description of the embodiments of the present invention, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0096] The terms "first," "second," etc., used in the embodiments of this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature.

[0097] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0098] Before demonstrating the embodiments, the following terms will be explained.

[0099] LITE-ORB (Oriented FAST and Rotated BRIEF) is an improved ORB feature detection method specifically optimized for tunnel scenes in low-light environments. It combines techniques such as grayscale correction, Freak descriptors, nonmaximum suppression, and feature point optimization to improve feature detection performance in low-light and low-texture scenes.

[0100] The oFAST algorithm is a feature point detection method based on the FAST (Features from Accelerated Segment Test) algorithm, used to quickly detect rotation-invariant feature points in images.

[0101] The FREAK (Fast Retina Keypoint) descriptor is an image feature descriptor inspired by the human visual system. It is a binary descriptor that generates a binary string by comparing the brightness of sampling points around the keypoint, thereby describing the keypoint's features. It features fast computation speed, small memory footprint, and strong robustness.

[0102] Non-Maximum Suppression (NMS) is a widely used algorithm in computer vision, primarily for object detection tasks, removing redundant bounding boxes to retain the most accurate detection results. The core idea of ​​NMS is to suppress non-maximum elements, that is, retaining the maximum value within the local neighborhood while suppressing other non-maximum elements (setting them to zero or discarding them). In edge detection (such as the Canny algorithm), NMS is used to refine edges, making them more accurate.

[0103] RANSAC (Random Sample Consensus) is an iterative method for estimating the parameters of a mathematical model on a dataset containing a large number of outliers. Its core idea is to fit a model to a randomly selected subset of data and then select the optimal model parameters by evaluating the consistency between the model and the data. The basic steps of the RANSAC algorithm are as follows: 1. Randomly select a subset of data, i.e., randomly select the minimum number of sample points from the dataset; these sample points are used to estimate the model parameters. 2. Use these sample points to estimate a model. 3. Substitute all data points into the estimated model and calculate the error between each point and the model. If the error is less than a preset threshold, the point is considered an "inlier"; otherwise, it is an "outlier." Count and record the number of inliers in the current model. Repeat the above steps multiple times, each time randomly selecting different sample points, until the model with the most inliers is found. Finally, select the model with the most inliers as the best model. The RANSAC algorithm is a powerful and robust estimation method, particularly suitable for data fitting problems with a large number of outliers. Through random sampling and iterative optimization, it can effectively extract the correct model parameters from noisy data and is widely used in fields such as computer vision, robot navigation and point cloud data processing.

[0104] The Viterbi algorithm is a dynamic programming algorithm used to find the sequence of hidden states in a Hidden Markov Model (HMM) that is most likely to generate the observed sequence. A second-order Hidden Markov Model (HMM2) is an extension of the HMM, considering second-order dependencies between hidden states; that is, the current state depends not only on the previous state but also on the previous two states. Hidden Markov Models are statistical models used to describe the transitions between hidden states at different time steps and the relationship between hidden states and observed data. They are widely used in fields such as speech recognition, natural language processing, and bioinformatics.

[0105] The V-Tunnel Algorithm is an image matching method based on the Viterbi algorithm and a second-order Hidden Markov Model (HMM2), specifically designed for visual localization in tunnel scenes. This model improves the accuracy and robustness of localization by utilizing visual features in tunnel scenes through sequence matching.

[0106] YOLOv5 is a deep learning-based object detection model. YOLOv5-LITE is a lightweight version of YOLOv5, primarily optimized for resource-constrained devices (such as embedded devices and mobile devices). Through a series of optimizations (such as removing the focus layer, reducing the number of high-channel C3 layers, and pruning the YOLOv5 head), YOLOv5-LITE significantly reduces the number of model parameters, computational cost, and memory usage compared to YOLOv5.

[0107] ASFF (Adaptive Spatial Feature Fusion) is an improved technique for multi-scale feature map fusion. By introducing a spatial attention mechanism, it dynamically adjusts the importance of features at different levels, thereby improving the performance of the model in object detection tasks.

[0108] The CBAM (Convolutional Block Attention Module) improves the attention mechanism by using Channel Attention (CAM) and Spatial Attention (SAM) modules to enhance the model's ability to extract key features. CAM learns the weights of each channel, making the network pay more attention to important channel features, while SAM learns the correlations between various locations in the feature map, enhancing attention to key regions.

[0109] The YOLO-TLO algorithm is a target detection and localization optimization algorithm based on YOLOv5, specifically designed for precise localization in tunnel scenarios. It combines image-level localization and metric-level localization, utilizing multi-frame image features and depth information to calculate the target's three-dimensional pose information.

[0110] The Levenberg-Marquardt (LM) algorithm is a nonlinear optimization algorithm that combines the advantages of gradient descent and the Gauss-Newton method. It dynamically adjusts the step size, ensuring both convergence stability and accelerating convergence speed. The core idea of ​​the LM algorithm is as follows: By introducing a dynamic parameter (such as a Lagrange multiplier), the step size is adjusted according to the dynamic changes of the objective function during iteration. Gradient descent is used in the early stages of iteration, and the Gauss-Newton method is used in the later stages. If the objective function value increases in the current iteration, it indicates that the step size is too large and needs to be reduced; if the objective function value decreases, it indicates that the step size is appropriate and can be increased to accelerate convergence.

[0111] This invention provides a visual positioning method, device, and medium for tunnel scenarios, which are described below.

[0112] Figure 1 This is a schematic flowchart of an embodiment of the visual positioning method for tunnel scenarios provided by the present invention, as shown below. Figure 1 As shown, the visual localization method for tunnel scenarios includes the following steps:

[0113] S100: Acquire image data of the tunnel area captured by the imaging component on the test vehicle; the image data includes color images and depth images.

[0114] S200: Obtain motion data collected by the inertial sensors on the test vehicle.

[0115] Specifically, the test vehicle model in the tunnel, such as Figure 2 As shown, the test vehicle is equipped with an imaging module and an IMU (Inertial Measurement Unit). The imaging module includes a depth camera and an RGB camera, while the inertial sensors include gyroscopes, accelerometers, etc. During the test vehicle's movement within the tunnel area, the RGB camera generates a color image by capturing the intensity of red, green, and blue light. The color image is represented by decomposing the color of each pixel into three basic components: red (R), green (G), and blue (B). The depth camera generates a depth image by measuring the distance between the object's surface and the camera. The IMU typically includes accelerometers, gyroscopes, and magnetometers, and collects motion data, including linear acceleration, angular velocity, and magnetic field direction.

[0116] S300: Based on the oFAST algorithm, FREAK algorithm, nonmaximum suppression algorithm and random sampling consensus algorithm, the feature points extracted from the color image are optimized to obtain a uniformly distributed set of feature points;

[0117] S400. Based on the Viterbi algorithm and the second-order hidden Markov model, a reference image sequence is selected according to the feature point set, the depth image and the motion data;

[0118] S500. Based on the improved target detection model, extract tunnel scene features from the color image;

[0119] S600. The current pose of the test vehicle is obtained based on the tunnel scene features, the reference image sequence, and the depth image.

[0120] Specifically, this invention utilizes the oFAST algorithm to rapidly detect feature points in color images, exhibiting rotation invariance and making it suitable for extracting key points in complex scenes such as tunnels. Based on the FREAK algorithm, it describes feature points, generating efficient binary descriptors to improve the speed and accuracy of feature matching. Based on the non-maximum suppression algorithm, it removes redundant feature points, resulting in a more uniform distribution and improved matching efficiency and robustness. Based on the random sampling consensus algorithm, it removes mismatched feature points through random sampling and consensus checks, further optimizing the resulting uniformly distributed feature point set. Furthermore, based on the Viterbi algorithm and a second-order hidden Markov model, it uses the feature point set, depth image, and motion data to select the most similar reference image sequence to the image to be matched. This effectively addresses the issues of repetitive and low-texture textures in tunnel scenes, improving matching accuracy and robustness. Furthermore, based on the improved target detection model combined with color and depth images, scene features (such as valves and doorways) in the tunnel are extracted, and the current pose of the test vehicle is calculated using reference image sequences and depth information. This can achieve centimeter-level positioning accuracy and is suitable for precise positioning in complex environments with low texture and repetitive texture, such as tunnels.

[0121] In some embodiments of the present invention, step S300 includes:

[0122] S310. Perform grayscale processing on the color image to obtain a grayscale image;

[0123] S320. Perform gamma correction processing on the grayscale image to obtain the processed image;

[0124] ;or,

[0125] S330. Normalize the grayscale image to obtain the processed image;

[0126] ;

[0127] in, To output grayscale values, The grayscale value of the input grayscale image. For compensation coefficient, Gamma coefficient, It is a positive number;

[0128] S340. Based on the multi-scale pyramid algorithm, multiple scale enhanced images are generated according to the processed image; the aspect ratio of the enhanced images at two adjacent scales is the same.

[0129] S350. Based on the oFAST algorithm, feature points are extracted from the enhanced image at multiple scales, and the orientation angle of the feature point is calculated according to the first coordinates corresponding to the feature point and the second coordinates corresponding to the centroid; the centroid is a point within a neighborhood circular region with the feature point as the center and the neighborhood radius as the radius.

[0130] , , ;

[0131] in, The neighborhood radius, In the enhanced image Moments within the range, , For order, , pixel coordinates of Power and Power of 1 For the first The grayscale value of each pixel Let the coordinates of the centroid of the moment be... It is the sum of the gray values ​​of all pixels in the neighborhood. For the grayscale value of a pixel in the neighborhood Weighted sum of distributions in different directions For the grayscale value of a pixel in the neighborhood Weighted sum of distributions in different directions;

[0132] S360. Generate a binary descriptor corresponding to each feature point based on the FREAK algorithm;

[0133] ;

[0134] ;

[0135] in, For binary descriptors, For the index variable in the summation process, The length of the binary code. For a sampling point pair, For sampling point pairs The pixel value of the previous sampling point, This is the pixel value of the next sampling point;

[0136] S370. Based on the nonmaximum suppression algorithm and the random sampling consensus algorithm, the feature points corresponding to the binary descriptor are optimized to obtain the feature point set.

[0137] Specifically, the LITE-ORB method mainly consists of the following parts: grayscale value correction, oFAST feature point detection, use of FREAK descriptor, non-maximum suppression, and feature point optimization. The ORB algorithm detects features by converting the image from RGB format to grayscale. While ordinary grayscale conversion is usable for images under normal lighting, it causes image information loss and distortion in low-light tunnel scenes. To preserve image information, this invention requires grayscale value correction. Gamma correction can be used, and its expression is shown in equation (1):

[0138] (1)

[0139] Since exponentiation can lead to an excessively large dynamic range in the output image, affecting its display, normalization is used to improve the gamma transform. Normalization can preserve detail information in the grayscale image. For 8-bit pixels, the expression is shown in equation (2):

[0140] (2)

[0141] To address the lack of scale invariance in the corner extraction of ordinary FAST, the ORB algorithm uses oFAST. oFAST uses a multi-scale pyramid to overcome the lack of scale invariance in FAST, and uses moments to determine the orientation of feature points, thus solving the problem of its lack of rotational invariance.

[0142] A multi-scale pyramid is a technique that scales a given image at a given scale to generate multiple layers of images at different scales. For the generated images, the aspect ratio between adjacent layers is the same, even though the dimensions are different. Then, the FAST algorithm is applied to filter feature points for images at multiple scales.

[0143] The oFAST algorithm uses the intensity centroid to determine the orientation of feature points, that is, it calculates the feature point orientation using... Let the centroid be the radius within which the feature point's coordinates converge. The vector from the feature point's coordinates to the centroid serves as the direction of the feature point. The moment is defined as: The coordinates of the centroid of the moment are: The direction of the feature points is: .in, , Used to construct moments of different orders, characterizing the distribution information of different dimensions of the image. Centroid coordinates middle, It is a normalization factor for calculating the centroid, ensuring that the centroid calculation is not affected by the overall grayscale scaling of the region; and These respectively represent the gray levels of the image. direction and Weighted sum of directional distributions. The value is obtained by calculating the centroid correlation moments, which is used to assign directional information to feature points, improve the robustness of feature matching in rotational scenarios, and make the features rotationally invariant.

[0144] This invention improves the ORB descriptor portion for tunnel scenarios by using a FREAK descriptor instead of a 256-bit binary descriptor. The FREAK descriptor's feature description formula is shown in equation (3):

[0145] (3)

[0146] in, The binary descriptor consists of a 1-bit differential Gaussian (DoG) sequence. It is the index variable used in the summation process to iterate from 0 to... The value of , The length of the binary code. This represents a pair of sampling points. The judgment formula is as shown in formula (4):

[0147] (4)

[0148] In this formula, Indicates sampling point pair The pixel value of the previous sampling point, This represents the pixel value of the next sampling point. FREAK will also obtain... The binary descriptors of bits are filtered and reduced in dimensionality to obtain better and more recognizable descriptors.

[0149] The detection results of ORB keypoints are often uneven. In order to make the detection results more regular and uniform to obtain better matching results, it is necessary to uniformize the distribution of points. This invention uses the nonmaximum suppression algorithm and the random sampling consensus algorithm to optimize the feature points corresponding to the binary descriptor to obtain the feature point set.

[0150] In some embodiments of the present invention, step S370 includes:

[0151] S371. Taking the feature point as the center, select its surrounding neighborhood. The average gray value of the feature point is calculated from each pixel.

[0152] ;

[0153] in, For the first The average gray value of each feature point;

[0154] S372. Based on the non-maximum suppression algorithm, non-maximum feature points are selected from a local region centered on the feature point and with a local region radius as the radius; the non-maximum feature point is the feature point in the local region whose average gray value is not the largest.

[0155] S373. The target feature point is obtained by performing preliminary optimization on the non-maximum feature point based on the suppression radius;

[0156] ;

[0157] in, The suppression radius is negatively correlated with the distinctiveness of the suppression region. Non-maximum feature points The corresponding Gaussian difference value, Non-maximum feature points The corresponding Gaussian difference value, The symbol for Euclidean distance calculation is... The constraint symbol indicates that conditions must be satisfied. Non-maximum feature points ;

[0158] S374. Based on the random sampling consensus algorithm, an inlier set is obtained according to the target feature points; the inlier set includes multiple pairs of inlier samples.

[0159] S375. Create a feature model based on the inlier sample, evaluate the remaining feature points based on the feature model, add the remaining feature points evaluated as inliers to the inlier set, update the feature model based on the inlier set, and iterate until the inlier ratio reaches the target to obtain the feature point set.

[0160] Specifically, for the feature points extracted in step S350, using Centered on these pixels, calculate this... The average gray value of each pixel is used as the gray value of the feature point, as shown in equation (5):

[0161] (5)

[0162] Set the radius of a local region across the entire image. The traditional ORB algorithm has a detection radius of 9 pixels, so the radius of the local region needs to be set to be greater than 9, generally above 20. Within a given local region, the maximum value within the region is determined by comparing the average gray values ​​of feature points. Feature points that are identified as non-maximum values ​​need to be suppressed, that is, ignored.

[0163] Although larger It can achieve a large distribution uniformity and regularity, but excessively large... Too many key points could lead to loss, affecting matching results. This invention designs an adaptive method to determine... The value: According to the traditional approach, Use it as an initial value, and then increase it. Assign values ​​until the desired number of keypoints is obtained. Although this can achieve a stable number of keypoints (partially determining the stable quality of image matching), it obviously generates a large amount of computation. Therefore, another approach is adopted: a smaller suppression radius is used in regions with more significant features to retain more keypoints, and correspondingly, a larger suppression radius is used in regions with less significant features. Based on this idea, by measuring the feature saliency of the region where the keypoint is located by the intensity of the neighboring pixels of the keypoint, an adaptive suppression radius algorithm can be obtained, as shown in Equation (6):

[0164] (6)

[0165] To adaptively calculate the suppression radius and make the feature points more evenly distributed: the suppression radius is small in areas with obvious features (such as areas with rich texture) to retain more key points; the suppression radius is large in areas with indistinct features (such as low-texture areas) to reduce redundant key points and improve the algorithm's adaptability to complex scenes such as tunnels. and It reflects the salience of the features.

[0166] After the initial detection of feature points, the target feature points selected according to equation (6) need to be optimized. The random sampling consensus algorithm, i.e., RANSAC, is used in this invention. First, the detected target feature points are screened, and N pairs of target feature points are selected as inliers. A feature model is fitted using these inliers. This feature model is suitable for the assumed inliers, and all unknown parameters can be calculated from the assumed inliers. Then, the inliers are used to estimate other points. If a point is suitable for the estimated feature model, it is considered to be an inlier and the inliers are expanded. The feature model is re-estimated using all the assumed inliers, because this feature model is only estimated based on the initial assumed inliers. After subsequent expansion, it needs to be updated. In this way, the feature model is continuously iterated and updated to evaluate the feature model until the best feature model is obtained. Suitable feature points can be screened based on the best feature model. These suitable feature points are gathered together to obtain the feature point set.

[0167] In some embodiments of the present invention, step S400 includes:

[0168] S410. Create a vehicle kinematic model of the test vehicle; the speed of the test vehicle is constant.

[0169] ;

[0170] in, For the test vehicle at time Location, For the test vehicle at time Location, For the test vehicle at time The position, the interval between two adjacent moments is less than a preset value;

[0171] S420. If the first acquisition frame rate of the imaging component is inconsistent with the second acquisition frame rate of the inertial sensor, the motion data is interpolated to ensure that the feature point set, the depth image and the motion data are time-aligned.

[0172] S430. After aligning the feature point set, the depth image, and the motion data with their poses, a subset of candidate nodes is obtained based on the real-time pose of the test vehicle.

[0173] S440. Based on the second-order hidden Markov model and the Viterbi algorithm, the reference image sequence corresponding to the undetermined observation sequence is obtained according to the candidate node subset.

[0174] Specifically, this invention abandons traditional single-frame matching and proposes the V-Tunnel Algorithm model. It uses the Viterbi algorithm and selects a sequence matching method based on Hidden Markov Model 2 (HMM2). This module includes three parts: vehicle motion model estimation, coarse matching based on pose information, and precise map sequence matching based on the HMM2 model. The data collection vehicle model in the tunnel is as follows... Figure 2 As shown. The vehicle is set to a constant speed, and the camera and IMU frame rates are determined. Here, the track car can be considered a simple model of uniform linear motion. Based on the kinematic model, the vehicle's position model is estimated using the following formula: In the formula, Indicates the current position. Indicates the position at the previous time step. The values ​​indicate the positions at the first two moments. It is important to note that the difference between different moments should not be too large. In the test scenario of this invention, it is necessary to ensure that the shape of the tunnel does not change significantly within a certain period of time, otherwise the movement of the vehicle will be affected by the curvature of the track.

[0175] To improve the robustness of the algorithm in this invention, a coarse matching procedure needs to be added before Hidden Markov Matching to ensure matching stability. This invention sets corresponding thresholds based on driving speed and camera frame rate. The specific steps of the coarse matching are as follows. In the experimental setup, it is necessary to ensure that the camera and IMU data are aligned. If frame rate discrepancies occur, interpolation processing is required. Ultimately, it is essential to ensure that each RGB image and depth map has corresponding IMU data.

[0176] After completing the pose alignment, it is necessary to calculate the coarse matching map nodes based on the current pose to be located, obtain the candidate node subset based on the real-time pose of the test vehicle, and then, based on the second-order hidden Markov model and the Viterbi algorithm, obtain the reference image sequence corresponding to the undetermined observation sequence based on the candidate node subset.

[0177] In some embodiments of the present invention, step S430 includes:

[0178] S431. Calculate the first distance value between the real-time pose and each map node in the map node set;

[0179] The nearest map node is selected from the set of map nodes, and the first distance value between the nearest map node and the real-time pose is the smallest.

[0180] S432. Select candidate map nodes to obtain the candidate node subset. The second distance value between the candidate map node and the nearest map node is less than a preset threshold. The candidate node subset includes the nearest map node and all the candidate map nodes.

[0181] Specifically, the current pose of the image to be matched is searched for on the map, and the map node closest to the current pose is calculated. The calculation formula is shown in equation (7):

[0182] (7)

[0183] in, The three-dimensional coordinates of the current pose to be located (i.e., the real-time pose of the track vehicle) represent the current position information of the system in space. These are the 3D coordinates of the map node. The nearest map node (the one with the smallest distance) is obtained by setting a threshold. , Using empirical values, and taking the pose of the nearest map node as the core, we continue to calculate the second distance between each candidate map node and the nearest map node. If so, the candidate map node will be added to the candidate node subset.

[0184] In some embodiments of the present invention, step S440 includes:

[0185] S441. Based on the Viterbi algorithm, the joint probability is obtained by recursively calculating the hidden state sequence and the undetermined observation sequence; the undetermined observation sequence is established based on the image data at the current time.

[0186] ;

[0187] S442. Calculate the maximum joint probability based on the posterior probability and the joint probability;

[0188] ;

[0189] in, For the maximum joint probability, For a moment The hidden state below, From time 1 to time 2 The observation sequence, Proportional to the sign, Let be the posterior probability, representing the probability given from time 1 to time 2. observation sequence Under the condition at time Hidden state The probability of occurrence For a moment The joint probability, representing time... Hidden state From time 1 to time 2 observation sequence The probability of them happening simultaneously;

[0190] S443. Calculate the initial state probability based on the position of the test vehicle;

[0191] , ;

[0192] in, This represents the map node index corresponding to the vehicle's actual starting position. It is a general index when traversing map nodes. This indicates that the test vehicle originated from a map node. The probability of departure. The initial state probability;

[0193] S444. Calculate the state transition probability based on the hidden state sequence and the position of the test vehicle on the map node;

[0194] ;

[0195] ;

[0196] in, , and All are map nodes. For the test vehicle at map nodes Location, The first covariance;

[0197] S444. The emission probability is calculated based on the image features and the number of feature points of the current image to be matched by the test vehicle at time t.

[0198] ;

[0199] in, The hidden state at time t Compared with observed values The number of feature points matched between, mean This is the number of ORB feature points obtained from the current image to be matched;

[0200] S445. Based on the chain rule and the properties of the second-order hidden Markov model, the forward probability is calculated according to the emission probability and the state transition probability.

[0201] ;

[0202] ;

[0203] in, For a moment The joint probability, For a moment The probability of launch, For a moment The state transition probability, For a moment The forward probability, For a moment The forward probability;

[0204] S446. Based on the posterior probability, the maximum joint probability, and the time interval... The forward probability is used to calculate the reference image sequence.

[0205] Specifically, in the HMM2-based model, the Viterbi algorithm treats each node in the candidate node subset as a state space in the coarse matching results. That is, state-space sequence The image features of the current image to be matched are used as observations. , This represents the time series of observations. The map node sequence from the coarse map matching result is used as the hidden state. , It is a time series of hidden states, hidden states Compared with observed values One-to-one correspondence. Therefore, the problem of precise image matching is transformed into the problem of finding the maximum joint probability. The principle of sequence matching is as follows: Figure 4 As shown, its expression is as shown in equation (8):

[0206] (8)

[0207] in, Indicates proportional to; It is the posterior probability, representing the probability given the time interval from time 1 to time 2. observation sequence Under the conditions, at any time System hidden state The probability of occurrence; It is the joint probability, representing time step. Hidden state From time 1 to time 2 observation sequence The probability of simultaneous occurrence reflects the joint distribution relationship between the hidden state and the observation sequence.

[0208] This invention proposes using the HMM2 model to model the transitions of the hidden states, that is... Only with and Related. In contrast, in a first-order HMM model... Only with Therefore, it's necessary to assume the current vehicle speed is constant, but this doesn't meet the actual needs of intelligent vehicles. The HMM2 model can predict the hidden state at the current moment through speed compensation, making it more suitable for real-world driving environments and more robust to changes in vehicle speed. Furthermore, compared to using a single map node as input for map matching, the HMM2 framework uses a sequence of map nodes as input, which improves the robustness and accuracy of map matching. This involves traversing all possible state sequences. Calculate joint probability As shown in equation (9):

[0209] (9)

[0210] In this invention, the Viterbi algorithm is used to recursively calculate... Similarly, is a joint probability, as in equation (8). To calculate recursion, let . According to the chain rule and the properties of the HMM2 model, its expression is shown in equation (10).

[0211] (10)

[0212] in, It's the probability of launch. It represents the state transition probability. Therefore, only the initial state probability needs to be determined to recursively calculate... As shown in equation (11):

[0213] (11)

[0214] The map nodes, i.e. the reference image sequence, can then be obtained by calculating the maximum joint probability, as shown in equation (12):

[0215] (12)

[0216] The steps for calculating the initial state probability, emission probability, and state transition probability in the Viterbi algorithm are as follows:

[0217] In this invention, the initial state probability refers to the probability distribution of the initial position of the test vehicle. Assume the vehicle starts from any point on the map. Start running, this is in the map node. state space The initial probability distribution is: And there are:

[0218] (13)

[0219] in, This represents the map node index corresponding to the vehicle's actual starting position. It is a general index used when traversing map nodes, used to represent any map node and determine whether that map node is the actual departure node of the vehicle. Indicates that the vehicle originates from a map node. The probability of departure.

[0220] In recursive computation, the state transition probability is: This indicates the state of the feature map from the first two time points. , Transition to the current state The probability of, where This indicates that the tunnel data acquisition vehicle started from a map node. go through Transferred to The probability of state transition between different map nodes in a tunnel needs to consider the kinematic model of the data acquisition vehicle and the structural characteristics of the tunnel. The former reflects the state transition relationship between different map node locations, while the latter reflects the connectivity of neighboring map nodes. , , The closer the three time points are, the higher the prediction accuracy and the greater the state transition probability. Based on the Gaussian model, the state transition probability is modeled using distance, and its expression is shown in equation (14):

[0221] (14)

[0222] in, Represents map nodes Location, mean It is based on and The obtained current state prediction, due to the kinematic model is It can be known that Using the first covariance To adjust the probability of state transitions between different map nodes, in this scenario, set... Then add all the necessary map nodes. After normalization, the data can be passed into the model. It can also be adjusted according to different sensors, as long as it conforms to the Gaussian model.

[0223] In this invention, the emission probability Indicates the state at the current moment. Obtaining observations The probability of emission. As can be seen from the previous steps, ORB descriptors are used for feature extraction to match the wall images of observation and state. For feature point optimization, RANSAC algorithm is used to remove low-quality or incorrect matches. The more feature points matched, the more similar the query image and the map image are, and the greater the emission probability. Therefore, the emission probability calculation in this invention is also based on the Gaussian formula, and its expression is shown in equation (15):

[0224] (15)

[0225] in, Indicates the current hidden state. Compared with observed values The number of ORB feature points matched between, mean This represents the number of ORB feature points acquired in the current image to be matched, which is the result of feature extraction using the LITE-ORB method in S1. For the image to be matched, an improved ORB feature extraction method consistent with that described above is used. First covariance. The method used to adjust the relationship between emission probabilities is set in this invention as follows: Finally, the emission probabilities at time t need to be normalized before being input into the model.

[0226] This model models the initial probability, state transition probability, and emission probability in a second-order Hidden Markov Model (HMM2) framework, and uses the Viterbi algorithm to recursively compute the optimal map matching sequence, setting a threshold to ensure sequence length. The map node with the highest joint probability is identified. This refers to the map node that is closest to the image to be matched.

[0227] In some embodiments of the present invention, step S500 includes:

[0228] S510. Add an attention mechanism and a spatial feature fusion mechanism to the Yolov5s network model to obtain an object detection model;

[0229] S520. The target detection model is trained according to the training set until the model parameters are determined to meet the standard according to the validation set to obtain the improved target detection module; both the training set and the validation set include image samples within the tunnel area, and the image samples are labeled with key features;

[0230] S530. Based on the YOLO algorithm and the improved target detection module, extract the tunnel scene features from the processed image obtained from the color image.

[0231] Specifically, the conventional Yolov5 model is suitable for general scenes, but its performance in target recognition, detection, and segmentation is not good in current low-light scenarios. This invention improves upon Yolov5s by proposing a Yolov5-LITE model, with the following improvements: ASFF improvement, CBAM attention mechanism improvement, and lightweighting improvement. This invention adds ASFF improvement to Yolov5s, with the following steps: 1. Input feature map. 2. Divide the feature map into blocks, into different regions. 3. Calculate adaptive weights at the region level and channel level for each region. 4. Fuse the feature map with these two levels of weights. 5. Repeat the above steps to achieve feature fusion at different levels. The principle is as follows: Figure 5 As shown. This invention, based on the CBAM attention mechanism, mainly consists of two parts: Channel Attention Mechanism (CAM) and Spatial Attention Mechanism (SAM). CAM is used to model the correlation between channels in the convolutional neural network. By learning the weights of each channel, the network can focus more on which channel features are more important. The input of CAM is a feature map, and the output is the weight of each channel. The features of different channels are weighted and summarized. SAM is used to learn the correlation between various positions in the feature map. It is obtained by average pooling the feature maps along the channel and spatial directions respectively, and combining these feature maps with task-related weights. Since this invention can be used for real-time detection, it is necessary to make lightweight improvements to the YOLO model to meet the needs of processors with different performance levels. This invention adds SimAM between the CAM channels in CBAM to enhance the response of the convolutional layer through the spatial context of the feature map, thereby capturing more detailed information in the image. This ensures that more features are extracted even when the image lighting is not good. SimAM utilizes the spatial context and uses the original receptive field in the convolutional kernel to enhance the convolutional layer.

[0232] In some embodiments of the present invention, step S600 includes:

[0233] S610. Select target feature images from the reference image sequence that have the same features as the tunnel scene;

[0234] S620. Filter the target depth image that is captured at the same time as the target feature map, and obtain the corresponding depth information based on the target depth image;

[0235] S630. Create a similar image set based on the reference image sequence; the similar image set includes multiple similar images, and the multiple similar images include a first target feature image and a second target feature image, wherein the first target feature image is the target feature image in the reference image sequence with complete tunnel scene features, and the second target feature image is the target feature image in the reference image sequence with incomplete tunnel scene features;

[0236] S640. Calculate the two-dimensional coordinates of the test vehicle based on the processed image and the similar image acquired at the current time, and calculate the current pose based on the two-dimensional coordinates and the depth information.

[0237] Specifically, step S400 yields an image-level localization image (i.e., the reference image sequence of this invention). This image-level localization image is obtained by using the HMM2 method to find the closest frame in the feature map based on the image to be matched. However, since the error of a single frame image is still above 10cm, falling within the decimeter-level localization range, metric-level localization is still needed to reduce the error. In the metric-level localization part, the sequence needs further expansion based on the reference image sequence, calculated based on feature points. This mainly consists of the following parts: In the metric-level localization stage, this invention needs to detect features in the image, generally including features such as: wall milestones, fire escape doors, station platforms, etc. A relevant model needs to be trained to detect these features. The map image is divided into three parts: two as the training set and one as the validation set, which improves detection accuracy. For the targets to be detected, the file is opened using labelme software, the edges of the features are drawn, labels are created, and saved in .json format to the directory where the image is located. After labeling, the dataset is exported in YOLO format adapted to the YOLO5-LITE model, thus completing the training set. The model can then be used to detect features, and the detection results are as follows: Figure 6 As shown.

[0238] The trained Yolov5-LITE model is used to detect image features, separately detecting the image to be matched and the image at the image-level localization level. Common features between the two are searched, and then filtered using a map. Images with the same features in the vicinity of the image-level results are extracted. An extraction threshold needs to be set. The approximate result is as follows. Figure 7 As shown. After selecting the matching sequence, feature extraction and matching of the image are required. The relative positions are calculated based on the image. The main steps are as follows:

[0239] Because image calculations are required to determine relative positions, camera calibration and calculation of camera intrinsic and extrinsic parameters are necessary. Based on previous depth camera data, depth information for the matching sequence is obtained. Depth images are typically represented as grayscale images, with each pixel's grayscale value corresponding to the distance or depth value at that location. Generally, darker pixels represent farther distances, while brighter pixels represent closer distances. Combined with IMU data, the three-dimensional coordinate information of these images can be obtained. Local feature detection is performed in the detected feature regions. This invention uses ORB local feature detection, setting the number of detected feature points to ensure both quantity and quality. Generally, the matching sequence consists of 15 images, with each sequence containing the same features.

[0240] In this invention, ORB is also used for feature matching, but unlike ordinary matching of two images, it is necessary to match every two images with complete features in the entire reference image sequence, retain all matchable point pairs, and then use images containing incomplete features for optimization to add some point pairs. After completing the matching of map nodes, it is matched with the features in the image to be matched. By using the positional relationship of the same features in multiple frames of images, the relative positional relationship between the image to be matched and all images in the matching sequence can be calculated. According to formula (16), the positional relationship between images can be calculated:

[0241] (16)

[0242] in, These are the two-dimensional pixel coordinates of feature points in the image to be matched. These are the two-dimensional pixel coordinates of the feature points corresponding to the sequence matching results, used to establish a correspondence with the feature points of the image to be matched and to calculate the relative pose.

[0243] After calculating the two-dimensional coordinates, the previously extracted depth information is added to obtain the preliminary three-dimensional pose information of the image to be matched.

[0244] After calculating the initial homography matrix using the above equation (16), it is necessary to use other feature point pairs to optimize the homography matrix. The LM algorithm is used to optimize the matrix. That is, to reduce the reprojection error and estimate whether the matrix parameters converge, a better matrix is ​​obtained. After optimization by multiple feature point pairs, the accurate pose information of the image to be matched can be obtained, and centimeter-level positioning can be achieved.

[0245] This application is based on Figure 3The illustrated process performs visual localization for tunnel scenes. This invention has several advantages: The lighting conditions in tunnel scenes are worse than in ordinary scenes, so classic ORB feature detection algorithms cannot effectively extract image features. This invention proposes the LITE-ORB method, which improves the following aspects: correcting image grayscale values, using FREAK descriptors instead of binary descriptors, adding non-maximum suppression, and using the RANSAC algorithm for optimization. These improvements optimize the ordinary ORB algorithm, significantly enhancing its feature detection capability in dimly lit scenes. Because tunnel scene features are similar, ordinary single-frame matching methods easily lead to errors where similar features are present but in different locations. This invention employs the Viterbi algorithm, based on a second-order Hidden Markov Model (HMM2). Leveraging the characteristic that the current time step of the HMM2 model is only related to the previous two time steps, the vehicle motion model is estimated, and the initial state probability matrix, state transition probability matrix, and emission probability matrix are calculated to achieve sequence matching. This method can improve matching accuracy and reduce the error rate. Since the conventional YOLOv5 model is not suitable for the low-light scenes described in this invention, we choose to improve the YOLOv5 model in three aspects: ASFF improvement, CBAM improvement, and lightweighting improvement, to enhance the model's performance and target detection capability in dim environments. For low-light scenes such as tunnels, the method of calculating the homography matrix by feature matching of only two frames of images cannot meet the accuracy requirements. This invention adopts a method of calculating pose using multi-frame image features. The YOLOv5-LITE model is used to extract image features, making full use of the scene's feature information, and feature point matching is performed simultaneously in the features of multiple frames of images to ensure the robustness of feature point pairs; depth map information acquired by a depth camera is also used to extract the three-dimensional information of the image, ensuring the three-dimensional input of the homography matrix; then, the multi-point pairing results are optimized using the LM algorithm to finally obtain the accurate three-dimensional pose information of the image to be matched.

[0246] To better implement the visual positioning method for tunnel scenarios in this embodiment of the invention, based on the visual positioning method for tunnel scenarios, the corresponding method is as follows: Figure 8 As shown, the present invention also provides an electronic device 800. The electronic device 800 includes a processor 801, a memory 802, and a display 803. Figure 8 Only some components of the electronic device 800 are shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead.

[0247] In some embodiments, processor 801 may be a central processing unit (CPU), microprocessor, or other data processing chip, used to run program code stored in memory 802 or process data, such as the visual positioning method for tunnel scenarios in this invention.

[0248] In some embodiments, processor 801 may be a single server or a group of servers. The server group may be centralized or distributed. In some embodiments, processor 801 may be local or remote. In some embodiments, processor 801 may be implemented on a cloud platform. In one embodiment, the cloud platform may include a private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, intranet, multi-cloud, etc., or any combination thereof.

[0249] In some embodiments, memory 802 may be an internal storage unit of electronic device 800, such as a hard disk or memory of electronic device 800. In other embodiments, memory 802 may also be an external storage device of electronic device 800, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on electronic device 800.

[0250] Furthermore, the memory 802 may include both internal storage units of the electronic device 800 and external storage devices. The memory 802 is used to store application software and various types of data installed on the electronic device 800.

[0251] In some embodiments, display 803 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 803 is used to display information from electronic device 800 and to display a visual user interface. Components 801-803 of electronic device 800 communicate with each other via a system bus.

[0252] In one embodiment, when the processor executes the personnel location detection program in memory 802, a visual positioning method for tunnel scenarios can be implemented, including:

[0253] Acquire image data of the tunnel area captured by the imaging component on the test vehicle; the image data includes color images and depth images;

[0254] Acquire motion data collected by the inertial sensors on the test vehicle;

[0255] Based on the oFAST algorithm, FREAK algorithm, nonmaximum suppression algorithm and random sampling consensus algorithm, the feature points extracted from the color image are optimized to obtain a uniformly distributed feature point set.

[0256] Based on the Viterbi algorithm and the second-order hidden Markov model, a reference image sequence is selected according to the feature point set, the depth image, and the motion data;

[0257] Based on the improved target detection model, tunnel scene features in the tunnel are extracted from the color image, and the current pose of the test vehicle is obtained based on the tunnel scene features, the reference image sequence, and the depth image.

[0258] It should be understood that when the processor 801 executes the personnel positioning and detection program in the memory 802, in addition to the functions mentioned above, it can also perform other functions, as can be found in the description of the corresponding method embodiments above.

[0259] Furthermore, this embodiment of the invention does not specifically limit the type of electronic device 800 mentioned. Electronic device 800 can be a mobile phone, tablet computer, personal digital assistant (PDA), wearable device, laptop computer, or other portable electronic device. Exemplary embodiments of portable electronic devices include, but are not limited to, portable electronic devices running iOS, Android, Microsoft, or other operating systems. The aforementioned portable electronic device can also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the invention, electronic device 800 may not be a portable electronic device, but rather a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0260] Accordingly, this application also provides a computer-readable storage medium for storing a computer-readable program or instruction. When the program or instruction is executed by a processor, it can implement the steps or functions of the visual positioning method for tunnel scenarios provided in the above-described method embodiments.

[0261] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.), and the computer program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0262] The foregoing has provided a detailed description of a visual positioning method, device, and medium for tunnel scenarios provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A visual positioning method for tunnel scenarios, characterized in that, include: Acquire image data of the tunnel area captured by the imaging component on the test vehicle; the image data includes color images and depth images; Acquire motion data collected by the inertial sensors on the test vehicle; Based on the oFAST algorithm, FREAK algorithm, nonmaximum suppression algorithm and random sampling consensus algorithm, the feature points extracted from the color image are optimized to obtain a uniformly distributed feature point set. Based on the Viterbi algorithm and the second-order hidden Markov model, a reference image sequence is selected according to the feature point set, the depth image, and the motion data; The reference image sequence, selected based on the Viterbi algorithm and a second-order hidden Markov model, according to the feature point set, the depth image, and the motion data, includes: Create a vehicle kinematic model for the test vehicle; the speed of the test vehicle is constant. ； in, For the test vehicle at time Location, For the test vehicle at time Location, For the test vehicle at time The position, the interval between two adjacent moments is less than a preset value; If the first acquisition frame rate of the imaging component is inconsistent with the second acquisition frame rate of the inertial sensor, the motion data is interpolated to ensure that the feature point set, the depth image and the motion data are time-aligned. After aligning the feature point set, the depth image, and the motion data in pose, a subset of candidate nodes is obtained based on the real-time pose of the test vehicle. The step of obtaining a subset of candidate nodes based on the real-time pose of the test vehicle includes: Calculate the first distance value between the real-time pose and each map node in the map node set; The nearest map node is selected from the set of map nodes, and the first distance value between the nearest map node and the real-time pose is the smallest. Candidate map nodes are selected to obtain the candidate node subset. The second distance value between the candidate map node and the nearest map node is less than a preset threshold. The candidate node subset includes the nearest map node and all the candidate map nodes. Based on the second-order hidden Markov model and the Viterbi algorithm, the reference image sequence corresponding to the undetermined observation sequence is obtained according to the candidate node subset; Based on the second-order hidden Markov model and the Viterbi algorithm, a reference image sequence corresponding to the undetermined observation sequence is obtained according to the candidate node subset, including: Based on the Viterbi algorithm, the joint probability is obtained by recursively calculating the hidden state sequence and the undetermined observation sequence; the undetermined observation sequence is established based on the image data at the current time. ； The maximum joint probability is calculated based on the posterior probability and the joint probability. ； in, For the maximum joint probability, For a moment The hidden state below, From time 1 to time 2 The observation sequence, Proportional to the sign, Let be the posterior probability, representing the probability from time 1 to time 2. observation sequence Under the condition at time Hidden state The probability of occurrence For a moment The joint probability, representing time step , Hidden state From time 1 to time 2 observation sequence The probability of them happening simultaneously From time 1 to time 2 Time series of hidden states observation sequence The joint probability of simultaneous occurrence; The initial state probability is calculated based on the position of the test vehicle; ， ； in, This represents the map node index corresponding to the vehicle's actual starting position. It is a general index when traversing map nodes. This indicates that the test vehicle originated from a map node. The probability of departure. The initial state probability; The state transition probability is calculated based on the hidden state sequence and the position of the test vehicle on the map node; ； ； in, , and All are map nodes. For the test vehicle at map nodes Location, The first covariance; The emission probability is calculated based on the image features and the number of feature points of the current image to be matched by the test vehicle at time t. ； in, The hidden state at time t Compared with observed values The number of feature points matched between, mean It is the number of ORB feature points obtained from the current image to be matched. It is the second covariance; Based on the chain rule and the properties of the second-order hidden Markov model, the forward probability is calculated according to the emission probability and the state transition probability. ； ； in, For a moment The joint probability, For a moment The probability of launch, For a moment The state transition probability, For a moment The forward probability, For a moment The forward probability; Based on the posterior probability, the maximum joint probability, and the time... The forward probability is used to calculate the reference image sequence; Based on the improved target detection model, tunnel scene features in the tunnel are extracted from the color image; The current pose of the test vehicle is obtained based on the tunnel scene features, the reference image sequence, and the depth image.

2. The visual positioning method for tunnel scenarios according to claim 1, characterized in that, The optimization process based on the oFAST algorithm, FREAK algorithm, nonmaximum suppression algorithm, and random sample consensus algorithm for feature points extracted from the color image to obtain a uniformly distributed feature point set includes: The color image is processed to obtain a grayscale image; The grayscale image is subjected to gamma correction to obtain the processed image; ;or, The grayscale image is normalized to obtain the processed image; ； in, To output grayscale values, The grayscale value of the input grayscale image. For compensation coefficient, Gamma coefficient, It is a positive number; Based on the multi-scale pyramid algorithm, enhanced images of multiple scales are generated from the processed image; the aspect ratio of the enhanced images at two adjacent scales is the same. Based on the oFAST algorithm, feature points are extracted from the enhanced image at multiple scales. The orientation angle of the feature point is calculated according to the first coordinates corresponding to the feature point and the second coordinates corresponding to the centroid. The centroid is a point within a neighborhood circular region with the feature point as the center and the neighborhood radius as the radius. ， ， ； in, The neighborhood radius, In the enhanced image Moments within the range, , For order, , pixel coordinates of power and Power of 1 For the first The grayscale value of each pixel Let the coordinates of the centroid of the moment be... It is the sum of the gray values ​​of all pixels in the neighborhood. For the grayscale value of a pixel in the neighborhood Weighted sum of distributions in different directions For the grayscale value of a pixel in the neighborhood Weighted sum of distributions in different directions; Generate a binary descriptor for each feature point based on the FREAK algorithm; ； ； in, For binary descriptors, For the index variable in the summation process, The length of the binary code. For a sampling point pair, For sampling point pairs The pixel value of the previous sampling point, This is the pixel value of the next sampling point; Based on the nonmaximum suppression algorithm and the random sampling consensus algorithm, the feature points corresponding to the binary descriptor are optimized to obtain the feature point set.

3. The visual positioning method for tunnel scenarios according to claim 2, characterized in that, The step of optimizing the feature points corresponding to the binary descriptor based on the nonmaximum suppression algorithm and the random sample consensus algorithm to obtain the feature point set includes: Centered on the feature point, select its surrounding neighborhood. The average gray value of the feature point is calculated from each pixel. ； in, For the first The average gray value of each feature point; Based on the non-maximum suppression algorithm, non-maximum feature points are selected from a local region centered on the feature point and with a local region radius as the radius; the non-maximum feature point is the feature point in the local region whose average gray value is not the largest. The target feature point is obtained by performing preliminary optimization on the non-maximum feature points based on the suppression radius; ； in, The suppression radius is negatively correlated with the degree of feature distinctness of the suppression region. Non-maximum feature points The corresponding Gaussian difference value, Non-maximum feature points The corresponding Gaussian difference value, The symbol for Euclidean distance calculation is... The constraint symbol indicates that conditions must be satisfied. Non-maximum feature points ; Based on the random sampling consensus algorithm, an inlier set is obtained according to the target feature points; the inlier set includes multiple pairs of inlier samples. A feature model is created based on the inlier samples. The remaining feature points are evaluated based on the feature model. The remaining feature points evaluated as inliers are added to the inlier set. The feature model is updated based on the inlier set. The update is iterated until the proportion of inliers reaches the target, and then the feature set is obtained.

4. The visual positioning method for tunnel scenarios according to claim 1, characterized in that, The step of extracting tunnel scene features from the color image based on the improved target detection model includes: An object detection model is obtained by adding an attention mechanism and a spatial feature fusion mechanism to the Yolov5s network model. The target detection model is trained on the training set until the model parameters are determined to meet the standards based on the validation set, thus obtaining the improved target detection module; both the training set and the validation set include image samples within the tunnel area, and the image samples are labeled with key features; Based on the YOLO algorithm and the improved target detection module, the tunnel scene features are extracted from the processed image obtained from the color image.

5. The visual positioning method for tunnel scenarios according to claim 4, characterized in that, The step of obtaining the current pose of the test vehicle based on the tunnel scene features, the reference image sequence, and the depth image includes: Select target feature images from the reference image sequence that have the same features as the tunnel scene; Filter the target depth image that is captured at the same time as the target feature map, and obtain the corresponding depth information based on the target depth image; A similar image set is created based on the reference image sequence; the similar image set includes multiple similar images, each of which includes a first target feature image and a second target feature image, wherein the first target feature image is the target feature image in the reference image sequence in which the tunnel scene features are complete, and the second target feature image is the target feature image in the reference image sequence in which the tunnel scene features are incomplete; The two-dimensional coordinates of the test vehicle are calculated based on the processed image and the similar image acquired at the current moment, and the current pose is calculated based on the two-dimensional coordinates and the depth information.

6. An electronic device, characterized in that, Including memory and processor, among which, The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps of the visual positioning method for tunnel scenes as described in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the visual positioning method for tunnel scenarios as described in any one of claims 1 to 5.

Images