A robust visual feature matching and detection method for underground parking
By constructing feature point and parking location detection models using UnsuperPoint and Center-net deep learning methods, the problem of low positioning accuracy caused by dim lighting and unclear texture in underground parking garage environments is solved, achieving high-precision automatic parking positioning and mapping.
Patent Information
- Application Number
- CN202210303258.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-24
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2042-03-24
AI Technical Summary
In underground parking garage environments, traditional visual SLAM systems suffer from low positioning accuracy due to dim lighting and unclear textures, making it impossible to accurately detect parking spaces and affecting the mapping quality of automated parking systems.
We employ deep learning methods based on UnsuperPoint and Center-Net to construct feature point detection and location detection models. Through offline training and feature point matching, we achieve robust visual feature matching and detection, adapting to environments with low lighting and indistinct textures.
In low-light conditions or when textures are not clearly visible, it improves positioning accuracy and mapping quality, reduces annotation costs, and achieves stability and accuracy in automatic parking.
Smart Images

Figure CN114782913B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of autonomous driving technology, and in particular to a robust visual feature matching and detection method for underground parking. Background Art
[0002] With the development of autonomous driving technology, automatic parking has become a current research hotspot. Automatic parking mainly uses sensors distributed throughout the vehicle itself and the surrounding environment to measure the relative distance, speed and angle between the vehicle itself and surrounding objects, and then calculates the operation process through the on-board computer platform or cloud computing platform, and controls the steering and acceleration and deceleration of the vehicle, so that the vehicle can automatically park in, out and some driving functions. The automatic parking process can generally include the following five major links: environmental perception, parking space detection and identification, parking path planning, parking path following, and control simulation display.
[0003] In actual applications, underground garages are a relatively simple environment. At the same time, reversing into the garage is also a key point that affects the user experience. Based on this, if you want to realize automatic parking in underground garages, it is essential to build a semantic map of underground garages. However, there are currently several difficulties in building maps for underground garages:
[0004] 1. Due to occlusion and shielding, the underground garage environment lacks GPS signals. Therefore, the positioning module must rely on a complete SLAM / LAM system.
[0005] 2. The texture is not obvious and the environment is dim, which will inevitably have a great impact on the traditional visual SLAM front-end system, and then affect the positioning accuracy, making the entire positioning system invalid and making it impossible to build a map.
[0006] 3. In an underground garage environment, whether the storage location can be accurately detected and the position can be restored will directly determine the quality of the subsequent construction of the semantic map, so the semantic information must be processed accurately. Summary of the invention
[0007] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a robust visual feature matching and detection method for underground parking, which can detect feature points in dim lighting or unclear texture, thereby achieving high-precision positioning and mapping, and further assisting the vehicle to complete automatic parking.
[0008] The purpose of the present invention can be achieved by the following technical solution: A robust visual feature matching and detection method for underground parking, comprising the following steps:
[0009] S1, collect the original RGB image to perform offline training on the feature point detection deep learning model to obtain a trained feature point detection model;
[0010] S2: Input the data to be tested into the feature point detection deep learning model, perform feature point detection, and obtain the corresponding feature point detection results;
[0011] S3: Input the RGB image to be tested and the true value label into the deep learning model for location detection for offline training to obtain a trained location detection model, wherein the RGB image to be tested is a bird's-eye view obtained by perspective transformation of the data to be tested;
[0012] S4: Input the RGB image to be tested into the deep learning model for storage location detection, perform storage location detection, and obtain the corresponding storage location detection result;
[0013] S5: Match the feature point detection result with the storage location detection result to obtain a matching result.
[0014] Furthermore, the original RGB image is specifically acquired by a monocular visible light camera, and the feature point detection deep learning model is specifically a neural network model based on the UnsuperPoint method.
[0015] Furthermore, the specific process of step S1 is as follows:
[0016] S11, collecting original RGB images;
[0017] S12, preprocessing the collected original RGB image to obtain a paired image;
[0018] S13, constructing a neural network model, inputting paired images into the neural network model for offline training, and obtaining a trained feature point detection model.
[0019] Furthermore, the step S12 specifically includes the following steps:
[0020] S121, input image I o Perform pre-transformation to generate the initial image I′ o , wherein the pre-transformation includes random horizontal flipping, vertical flipping and random cropping, wherein the random cropping is to randomly scale the image and crop it to a fixed value to ensure batch training;
[0021] S122, for the initial image I′ o Perform two sets of operations separately to obtain an image pair containing two images and the transformation matrix H between the image pairs k , wherein the first group of operations in the two groups of operations is to perform non-perspective transformation on the image, and the second group of operations is to perform perspective transformation and non-perspective transformation on the image, and the specific operation steps of the non-perspective transformation of the first group of operations and the second group of operations are different;
[0022] The non-perspective transformation is to adjust the image hue, etc., including color tremor, motion blur, Gaussian noise, random illumination contrast transformation, random atomization, Gamma transformation, channel aliasing, these transformations are operated in series, and each transformation is used according to the set probability;
[0023] The perspective transformation is to randomly select four points (a1, b1, c1, d1) in the upper left, upper right, lower left and lower right corners of the image, and let these four points form matching point pairs (a1-P1, b1-P2, c1-P3, d1-P4) with the upper left, upper right, lower left and lower right corner points (P1, P2, P3, P4) respectively, so as to calculate the transformation matrix H k , and apply this transformation to the image to obtain the image after the perspective transformation;
[0024] S123, post-processing the image pair, splicing the processed images in the channel dimension to obtain (o′1, o′2), wherein the post-processing specifically converts the image into a grayscale image, and then uses the sober_x and sober_y operators respectively to generate an x-direction gradient map and a y-direction gradient map, normalize them to 1, and merge them with the original image.
[0025] Furthermore, the transformation matrix H k The specific calculation process is:
[0026] Let H k The matrix form is:
[0027]
[0028] Taking the matching point pair a1-P1 as an example, let the coordinates of a1 on the image be [x1, y1, 1] T , the coordinates of P1 on the image are [X1, Y1, 1] T , the image homography transformation relationship is:
[0029] P1=H k a1
[0030] For matrix operations, we have the equation:
[0031]
[0032]
[0033] One matching point pair can form two equations. By arranging the eight equations formed by all four matching point pairs, we can get a new equivalent matrix calculation form:
[0034] MH′=N
[0035]
[0036] H′=[h 11 h 12 h 13 h 21 h 22 h 23 h 31 h 32 ] T
[0037] N=[X1 Y1 X2 Y2 X3 Y3 X4 Y4] T
[0038] Where [x1, y1] T is the coordinate of a1 on the image, [X1, Y1] T is the coordinate of P1 on the image, and so on, [x4, y4] T is the coordinate of d1 on the image, [X4, Y4] T is the coordinate of P4 on the image;
[0039] From the equivalent matrix calculation form, we have:
[0040] H′=M -1 N
[0041] The transformation matrix H can be solved k The parameters of the transformation matrix H can be solved k .
[0042] Furthermore, the neural network model constructed in step S13 includes a backbone network, which is used to perform the joint task of point confidence estimation, point coordinate regression and descriptor extraction. The backbone network is divided into two branches: one branch is used to process the original image, and the other branch is used to process the image transformed by the transformation matrix. The extracted points are projected into the same image coordinate system through the true value of the transformation matrix, and the point distance of each pair is calculated. The point pairs with a distance less than a threshold value Dis are taken as valid point pairs, and point correspondences are constructed to perform self-supervised learning;
[0043] The learning loss function of the neural network model is specifically:
[0044] Loss1=α Sco L sco +α Dis L Dis +α Usp L Usp +α Reg L Reg +α Des L Des +α Dec LDec
[0045]
[0046] L Reg =L Regx +L Regy
[0047]
[0048]
[0049]
[0050] L Dec =related_coefficient_sum(des 0 )+related_coefficient_sum(des 1 )
[0051] Among them, Loss1 is the total loss function of the feature point detection deep learning model; L Sco is the point confidence loss, which is represented by the square of the difference in scores between two points. Sco For L sco The corresponding weight; L Dis is the Euclidean distance loss of the point pair, α Dis For L Dis The corresponding weight; L Usp is the repeatability loss based on the point-to-point distance, s i is the confidence of the point pair, d i is the distance between the points with index i, is the mean distance of all point pairs, α Usp For L Usp The corresponding weight; L Reg is the regularization loss of the relative distribution uniformity of the points, out is the position input corresponding to the grid area in the image where the feature point is located, the subscripts x and y represent the relative components of the two corresponding positions, ranging from 0 to 1, the superscripts 1 and 2 correspond to the first and second images respectively, o represents the oth feature, argsort represents the index position after sorting its output, α Reg For L Reg The corresponding weight; L Desis the consistency loss of the descriptor, t is a hyperparameter used to control the penalty strength, the superscripts 1 and 2 of des represent the first and second images, i and k / k' represent the i-th descriptor of the first image and the k / k'-th descriptor of the second image, the subscript o / o' of des represents that this is the o / o'-th descriptor of the associated point set, n1 represents the total number of descriptor pairs, the distance between each pair of descriptors in the numerator is less than 8 pixels (corresponding to the i, k point pair), and the distance in the denominator is greater than 8 (corresponding to the i, k' point pair), α Des For L Des The corresponding weight; L Dec To describe the correlation loss of the sub-element, the related_coefficient_sum function calculates the correlation coefficients between different dimensions and sums them up. Dec YesL Dec The corresponding weights;
[0052] The image pair (o′1, o′2) is input into the constructed neural network model, and the corresponding feature points are obtained after inputting o′1, including its score Location And the descriptor The superscript 1 represents the first image o′1, and the subscript k represents the kth feature point; similarly, after inputting o′2, the corresponding
[0053] The score and descriptor are the outputs of the corresponding grid positions, and the position of the feature point Then it is necessary to calculate the position output parameters of the network to obtain;
[0054] The score, position, descriptor, and grid are specifically:
[0055] After the image is input into the network, the corresponding outputs are obtained from the network's score output head, position output head, and descriptor output head. Assuming that the input image size is (h, w), the sizes of the score output head, position output head, and descriptor output head are (1, h / 8, w / 8), (2, h / 8, w / 8), and (256, h / 8, w / 8), respectively. The first one in the above brackets is the number of channels, and the last two are the height and width of the output result respectively.
[0056] Assume that the size of the input image is 24*24, the output image is divided into 3*3 squares of size 8*8. Each square will have a score, two position parameters and a 256-dimensional descriptor, corresponding to the outputs of (1, h / 8, w / 8), (2, h / 8, w / 8), and (256, h / 8, w / 8). The score of the feature point of the square is the output of the score output head corresponding to the square. Each grid has only one feature point, and whether it will be used will eventually be determined based on the size of its score. The position of the square is determined by the two channels corresponding to the position output head. The parameters output by these two channels are in the range of 0-1, corresponding to the relative horizontal and vertical positions in the grid. The descriptor of the square is determined by the 256-dimensional vector of the corresponding descriptor branch. Similarly, the feature points of each square output, that is, its score, position and descriptor, can be obtained.
[0057] Furthermore, the data to be measured is specifically a front view image of the vehicle taken by a monocular camera; the perspective transformation is specifically performed by calculating the homography matrix Apply this homography transformation to the front view image to obtain a bird's-eye view after perspective transformation. The homography matrix is Specifically:
[0058]
[0059] n=[0 0 -1] T
[0060] Among them, K1 is the internal parameter of the non-bird's-eye view camera, and K2 is the internal parameter of the bird's-eye view virtual camera. is the camera external parameter, and d is the height of the virtual bird's-eye view camera.
[0061] Furthermore, the true value label in step S3 is obtained by making a corner point. The specific process is: let the corner point be p i , the subscript i represents the i-th corner point, and is drawn on a graph of size (h / 4, w / 4) with p i / 4 is a local Gaussian distribution centered on the integer part, and the resulting graph is the true value label feature graph Y.
[0062] Furthermore, the deep learning model for location detection in step S3 is specifically a neural network model based on the Center-net method: using the Hourglass network as the feature extraction layer, there are two output heads, one is a single-channel heat map, which is used to detect the area where the corner point may be located; the other is a dual-channel relative position map, which outputs the deviation to restore the precise position;
[0063] The loss function of the location detection deep learning model is:
[0064] Loss2=w1L seg+w2L off
[0065]
[0066]
[0067]
[0068] Among them, Loss2 is the total loss function of the deep learning model for location detection; L seg is the corner probability loss, Y is the true value label feature map, is a single-channel feature map output by the network. The subscripts x and y are the indices in the corresponding feature map. α and β are hyperparameters. xy =1, 1 no slots The value is 0, otherwise it is 1, N is the number of corner points, w1 is L seg The corresponding weight, w2 is L off The corresponding weight; L off is the deviation regression loss, the wavy line represents rounding down, O is the dual-channel feature map output by the network, p is the corner point, and k is the index of p and O.
[0069] Furthermore, the specific process of obtaining the storage location detection result in step S4 is: screening according to the heat map output by the storage location detection neural network, selecting points on the map that are greater than the score threshold as candidate points, and then performing non-maximum suppression (NMS) to obtain storage location candidate points, the result is:
[0070] p out =4p seg +p off
[0071] Among them, p seg is the position of the heat map where the score is greater than the threshold, p off It is the residual result of the corresponding dual-channel image output.
[0072] Furthermore, the step S5 specifically includes the following steps:
[0073] S51, inversely transform the storage location corner points of the storage location detection result to the front view to generate the front view corner points. The specific formula of the inverse transformation is:
[0074]
[0075] in, is the corner point of the storage location, is the homography matrix, is the front view corner point;
[0076] S52, according to the front view corner point The descriptor at the corner point of the front view on the feature map is obtained to complete the matching. The descriptor d at the corner point of the front view on the feature map is slot The coordinates are:
[0077]
[0078]
[0079] in, and They are The x and y coordinates of the numpy array are rounded down.
[0080] After obtaining the block where the front view corner point is located, we can get its corresponding descriptor, that is,
[0081] d slot =des x,y
[0082] Using this descriptor for matching can well correlate the corner points of the warehouse locations between subsequent different frames.
[0083] Compared with the prior art, the present invention constructs a feature point detection model and a storage location detection model, and uses the feature point detection results and the storage location detection results for matching, thereby being able to stably extract and match feature points under conditions such as illumination changes, thereby improving the illumination robustness of image matching, and being able to accurately detect feature points in dim lighting or unclear textures, thereby achieving high-precision positioning and mapping;
[0084] The present invention realizes an unsupervised illumination robust feature point detection method, which does not require a lot of labeling work and greatly reduces the cost;
[0085] The present invention establishes new loss functions for the feature point detection model and the storage location detection model respectively. When applied in the deep learning model, it can more effectively detect feature points in dim lighting or unclear texture conditions, thereby achieving more accurate positioning and mapping. BRIEF DESCRIPTION OF THE DRAWINGS
[0086] Figure 1 It is a schematic diagram of the method flow of the present invention;
[0087] Figure 2 Schematic diagram of the preprocessing process of input data during feature point detection in the embodiment;
[0088] Figure 3 Schematic diagram of non-viewing angle transformation of input data during feature point detection in an embodiment;
[0089] Figure 4Schematic diagram of multiple point matching during perspective transformation of input data during feature point detection in an embodiment;
[0090] Figure 5 Schematic diagram of post-processing process of input data during feature point detection in the embodiment;
[0091] Figure 6 A schematic diagram of the architecture of a deep learning network for feature point detection in an embodiment;
[0092] Figure 7 Schematic diagram of input and output of a deep learning network for feature point detection in an embodiment;
[0093] Figure 8 Schematic diagram of bird's-eye view transformation during corner point detection in the embodiment;
[0094] Fig. 9 A schematic diagram of a local two-dimensional Gaussian distribution of true value labels required for training a deep learning model for storage location detection generated in an embodiment;
[0095] Fig.10 A schematic diagram of performing an inverse transformation of the storage location corner points onto the front view to generate the front view corner points in the embodiment;
[0096] Fig.11 This is a schematic diagram of associating storage location corner points between subsequent different frames in an embodiment. DETAILED DESCRIPTION
[0097] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.
[0098] Example
[0099] like Figure 1 As shown, a robust visual feature matching and detection method for underground parking includes the following steps:
[0100] S1, collect the original RGB image to perform offline training on the feature point detection deep learning model to obtain a trained feature point detection model;
[0101] In this embodiment, a neural network model based on the UnsuperPoint method is used for offline training. The specific process of offline training is as follows:
[0102] S1.1: Preprocess the collected original RGB image (such as Figure 2 As shown), to obtain paired images, wherein the original RGB image is collected by a monocular visible light camera, and the preprocessing includes the following steps:
[0103] S1.1.1: For the input image I o Perform pre-transformation to generate the initial image I′ oThe pre-transformation operations include random horizontal flipping, vertical flipping, and random cropping. Random cropping refers to randomly scaling and cropping the image to a fixed value to ensure batch training;
[0104] S1.1.2: For the initial image I′ o Perform two sets of operations separately to obtain an image pair (i.e., two images), and at the same time obtain the transformation matrix H between the image pairs k , where the first group of operations is to perform non-perspective transformation on the image, and the second group of operations is to perform perspective transformation and non-perspective transformation on the image, and the non-perspective transformation (such as Figure 3 As shown in the figure, the image hue is adjusted, including color tremor, motion blur, Gaussian noise, random illumination contrast change, random fogging, Gamma change, channel aliasing, etc. These changes are operated in series, and each change has a certain probability of being used. The perspective change is to randomly select four points (a1, b1, c1, d1) in the upper left, upper right, lower left, and lower right corners of the image, and let these four points form matching point pairs (a1-P1, b1-P2, c1-P3, d1-P4) with the upper left, upper right, lower left, and lower right corner points (P1, P2, P3, P4) respectively. Figure 4 As shown), the homography matrix H is calculated k , and apply this transformation to the image to obtain the image after the perspective transformation, wherein the specific operation steps of the first group of operations and the non-perspective transformation of the second group of operations are different;
[0105] S1.1.3: Post-process the image pairs, such as Figure 5 As shown, the image is converted into a grayscale image and then the sober_x and sober_y operators are used respectively to generate the x-direction gradient map and the y-direction gradient map, which are normalized to 1 and merged with the original image. The processed images are spliced in the channel dimension to obtain (o′1, o′2).
[0106] S1.2: Build a neural network model, input the paired images into the neural network model for offline training, and obtain a trained matching model. The neural network model architecture is as follows Figure 6 As shown, it includes a backbone network, which is used to perform the joint tasks of point confidence estimation, point coordinate regression and descriptor extraction. The backbone network is divided into two branches: one branch is used to process the original image, and the other branch is used to process the image after the homography matrix transformation. The extracted points are projected into the same image coordinate system through the true value of the homography matrix, and the point distance of each pair is calculated. The point pairs with a distance less than the threshold Dis are taken as valid point pairs, and the point correspondence relationship is constructed for self-supervised learning. The learning loss function of the neural network model is specifically:
[0107] Loss1=α ScoL Sco +α Dis L Dis +α Usp L Usp +α Reg L Reg +α Des L Des +α Dec L Dec
[0108] Among them, Loss1 is the total loss function of the feature point detection deep learning model;
[0109] L sco is the point confidence loss, which is represented by the square of the difference in scores between two points. Sco For L sco The corresponding weights;
[0110] L Dis is the Euclidean distance loss of the point pair, α Dis For L Dis The corresponding weights;
[0111] L Usp is the repeatability loss based on the point-to-point distance, expressed as:
[0112]
[0113] where s i is the confidence of the point pair, d i is the distance between the points with index i, is the mean distance of all point pairs, α Usp For L Usp The corresponding weights;
[0114] L Reg is the regularization loss of the relative distribution uniformity of the points, and the expression is:
[0115] L Reg =L Regx +L Regy
[0116] in:
[0117]
[0118]
[0119] Where out is the position input corresponding to the grid area in the image where the feature point is located, the subscripts x and y represent the relative components of the two corresponding positions, and their range is 0-1. The superscripts 1 and 2 correspond to the first and second images, o represents the oth feature, argsort represents the index position after sorting its output, and α Reg For L Reg The corresponding weights;
[0120] L Des is the consistency loss of the descriptor, expressed as:
[0121]
[0122] Where t is a hyperparameter used to control the penalty strength, the superscripts 1 and 2 of des represent the first and second images, i and k / k' represent the i-th descriptor of the first image and the k / k'th descriptor of the second image, the subscript o / o' of des represents that this is the o / o'th descriptor of the associated point set, n1 represents the total number of descriptor pairs, where the distance between each pair of descriptors in the numerator is less than 8 pixels (corresponding to the i, k point pair), and the distance in the denominator is greater than 8 (corresponding to the i, k' point pair), α Des For L Des The corresponding weights;
[0123] L Dec is the correlation loss of the descriptor, expressed as:
[0124] L Dec =related_coefficient_sum(des 0 )+related_coefficient_sum(des 1 )
[0125] The related_coefficient_sum function calculates the correlation coefficients between different dimensions and sums them up. Dec YesL Dec The corresponding weights.
[0126] S2: Input the data to be tested into the feature point detection deep learning model to perform feature point detection;
[0127] like Figure 7As shown, assuming that the size of the input image is 24*24, the output image is divided into 3*3 squares of size 8*8. Each square will have a score, two position parameters and a 256-dimensional descriptor, corresponding to the outputs of (1, h / 8, w / 8), (2, h / 8, w / 8), and (256, h / 8, w / 8). The score of the feature point of the square is the output of the score output head corresponding to the square (each square has only one feature point, and whether it will be used will eventually be determined based on the size of its score). The position of the square is determined by the two channels corresponding to the position output head. The parameters output by these two channels are in the range of 0-1, corresponding to the relative horizontal and vertical positions in the grid. The descriptor of the square is determined by the 256-dimensional vector of the corresponding descriptor branch. Similarly, the feature points output by each square can be obtained, that is, its score, position and descriptor;
[0128] S3: Input the RGB image and the true value label into the deep learning model for location detection for offline training to obtain a trained location detection model;
[0129] The RGB image is a bird's-eye view obtained by perspective transformation of the front view image taken by the monocular camera. The perspective transformation is obtained by calculating the homography matrix Applying this transformation to the front view image gives a bird's-eye view after the perspective transformation, such as Figure 8 As shown. Homography matrix It is calculated by the following method:
[0130]
[0131] Among them, K1 is the internal parameter of the non-bird's-eye view camera, and K2 is the internal parameter of the bird's-eye view virtual camera. It is assumed that the two are consistent, so they have the same signs. is the camera external parameter, d is the virtual bird's-eye view camera height, n=[0 0 -1] T .
[0132] The true value label is made by the corner point. The specific process of making the true value label by the corner point is: let the corner point be p i , the subscript i represents the i-th corner point. On a graph of size (h / 4, w / 4) i The local Gaussian distribution centered on the integer part of / 4 is obtained as the true value label feature map Y, such as Fig. 9 shown.
[0133] In this embodiment, the deep learning model for location detection is a neural network model based on the Center-net method, using the Hourglass network as the feature extraction layer, with two output heads, one is a single-channel heat map, used to detect the area where the corner point may be located; the other is a dual-channel relative position map, which outputs the deviation to restore the precise position. The loss function of the neural network model is:
[0134] Loss2=w1L seg +w2L off
[0135] Among them, Loss2 is the total loss function of the deep learning model for location detection;
[0136] L seg is the corner point probability loss, expressed as:
[0137]
[0138] Where Y is the true value label feature map. is a single-channel feature map output by the network. The subscripts x and y are the indices in the corresponding feature map. α and β are hyperparameters. xy =1, 1 no slots The value is 0, otherwise it is 1, N is the number of corner points, w1 is L seg The corresponding weight, w2 is L off The corresponding weights;
[0139] L off is the deviation regression loss, expressed as:
[0140]
[0141] in
[0142]
[0143] The wavy line represents rounding down, O is the dual-channel feature map output by the network, p is the corner point, and k is the index of p and O.
[0144] S4: Input the data to be tested into the deep learning model for location detection to perform location detection;
[0145] The data to be tested is a bird's-eye view obtained by perspective transformation of the front view image taken by the monocular camera of the vehicle, and the transformation principle is the same as the aforementioned bird's-eye view transformation principle.
[0146] After the data is input into the warehouse location detection neural network model, it is screened according to the heat map output by the neural network, and the points on the map that are greater than the score threshold are selected as candidate points. Then non-maximum suppression (NMS) is performed to obtain the warehouse location candidate points. The result is:
[0147] p out =4p seg +p off
[0148] where p seg is the position of the heat map where the score is greater than the threshold, p off It is the residual result of the corresponding dual-channel image output.
[0149] S5: Match the feature point detection results with the storage location detection results. Specifically:
[0150] S5.1: Perform an inverse transformation on the storage location corner points detected in step S4 to the front view to generate the front view corner points, such as Fig.10 As shown;
[0151] Set storage corner point Perform inverse transformation to get the front view corner points The formula is:
[0152]
[0153] This is the homography matrix for converting the front view to the bird's eye view as described above.
[0154] S5.2: Through the process of step S2, the descriptor at the front view corner point position on the feature map is obtained, and the matching is completed. Set the front view corner point Then its corresponding descriptor d slot The coordinates are:
[0155]
[0156]
[0157] in and yes The x and y coordinates of the front view corner point are obtained, and the corresponding descriptor can be obtained, which is
[0158] d slot =des x,y
[0159] Using this descriptor for matching can well correlate the corner points of the warehouse locations between subsequent frames, such as Fig.11 shown.
[0160] In summary, the adoption of this technical solution can stably extract and match feature points under conditions such as lighting changes, thereby improving the lighting robustness of image matching. In practical applications, it provides an incremental learning method for visual LAM in dim and texture-missing environments, enabling the front-end framework of LAM to move towards a data-driven learning model.
Claims
1. A robust visual feature matching and detection method for underground parking, characterized in that: The following steps are involved: S1, collect the original RGB image to perform offline training on the feature point detection deep learning model to obtain a trained feature point detection model; S2: Input the data to be tested into the feature point detection deep learning model, perform feature point detection, and obtain the corresponding feature point detection results; S3: Input the RGB image to be tested and the true value label into the deep learning model for location detection for offline training to obtain a trained location detection model, wherein the RGB image to be tested is a bird's-eye view obtained by perspective transformation of the data to be tested; S4: Input the RGB image to be tested into the deep learning model for storage location detection, perform storage location detection, and obtain the corresponding storage location detection result; S5: Match the feature point detection result with the storage location detection result to obtain a matching result; The original RGB image is specifically acquired by a monocular visible light camera, and the feature point detection deep learning model is specifically a neural network model based on the UnsuperPoint method. The specific process of step S1 is: S11, collecting original RGB images; S12, preprocessing the collected original RGB image to obtain a paired image; S13, constructing a neural network model, inputting paired images into the neural network model for offline training, and obtaining a trained feature point detection model; Step S12 specifically includes the following steps: S121, input image I o Perform pre-transformation to generate the initial image I′ o , wherein the pre-transformation includes random horizontal flipping, vertical flipping and random cropping, wherein the random cropping is to randomly scale the image and crop it to a fixed value to ensure batch training; S122, for the initial image I′ o Perform two sets of operations separately to obtain an image pair containing two images and the transformation matrix H between the image pairs k , wherein the first group of operations in the two groups of operations is to perform non-perspective transformation on the image, and the second group of operations is to perform perspective transformation and non-perspective transformation on the image, and the specific operation steps of the non-perspective transformation of the first group of operations and the second group of operations are different; The non-perspective transformation is to adjust the image hue, including color tremor, motion blur, Gaussian noise, random illumination contrast transformation, random atomization, Gamma transformation, channel aliasing, these transformations are operated in series, and each transformation is used according to the set probability; The perspective transformation is to randomly select four points (a1, b1, c1, d1) in the upper left, upper right, lower left and lower right corners of the image, and let these four points form matching point pairs (a1-P1, b1-P2, c1-P3, d1-P4) with the upper left, upper right, lower left and lower right corner points (P1, P2, P3, P4) respectively, so as to calculate the transformation matrix H k , and apply this transformation to the image to obtain the image after the perspective transformation; S123, post-processing the image pair, splicing the processed images in the channel dimension to obtain (o′1, o′2), wherein the post-processing specifically converts the image into a grayscale image, and then uses sober_x and sober_y operators to generate an x-direction gradient map and a y-direction gradient map, normalizes them to 1, and merges them with the original image; The transformation matrix H k The specific calculation process is: Let H k The matrix form is: Taking the matching point pair a1-P1 as an example, let the coordinates of a1 on the image be [x1, y1, 1] T , the coordinates of P1 on the image are [X1,Y1,1] T , the image homography transformation relationship is: P1=H k a1 For matrix operations, we have the equation: One matching point pair can form two equations. By arranging the eight equations formed by all four matching point pairs, we can get a new equivalent matrix calculation form: MH′=N H′=[h 11 h 12 h 13 h 21 h 22 h 23 h 31 h ∑2 ] T <h2 style=";text-align:left;direction:ltr">N=[X1 Y1 X2 Y2 X3 Y3 X4 Y4]<h2 style=";text-align:left;direction:ltr"> T Where [x1,y1] T is the coordinate of a1 on the image, [X1, Y1] T is the coordinate of P1 on the image, and so on, [x4, y4] T is the coordinate of d1 on the image, [X4, Y4] T is the coordinate of P4 on the image; From the equivalent matrix calculation form, we have: H′=M -1 N The transformation matrix H can be solved k The parameters of the transformation matrix H can be solved k ; The neural network model constructed in step S13 includes a backbone network, which is used to perform the joint task of point confidence estimation, point coordinate regression and descriptor extraction. The backbone network is divided into two branches: one branch is used to process the original image, and the other branch is used to process the image transformed by the transformation matrix. The extracted points are projected into the same image coordinate system through the true value of the transformation matrix, and the point distance of each pair is calculated. The point pairs with a distance less than the threshold Dis are taken as valid point pairs, and the point correspondence relationship is constructed to perform self-supervised learning; The learning loss function of the neural network model is specifically: Loss1=a Sco L Sco +a Dis L Dis +a Usp L Usp +a Reg L Reg +a Des L Des +a Dec L Dec L Reg =L Regx +L Regy L Dec =related_coefficient_sum(des 0 )+realted_coefficient_sum(des 1 ) Among them, Loss1 is the total loss function of the feature point detection deep learning model; L Sco is the point confidence loss, which is represented by the square of the difference in scores between two points. Sco For L Sco The corresponding weight; L Dis is the Euclidean distance loss of the point pair, α Dis For L Dis The corresponding weight; K Usp is the repeatability loss based on the point-to-point distance, s i is the confidence of the point pair, d i is the distance between the points with index i, is the mean distance of all point pairs, α Usp For L Usp The corresponding weight; L Reg is the regularization loss of the relative distribution uniformity of the points, out is the position input corresponding to the grid area in the image where the feature point is located, the subscripts x and y represent the relative components of the two corresponding positions, ranging from 0 to 1, the superscripts 1 and 2 correspond to the first and second images respectively, o represents the oth feature, argsort represents the index position after sorting its output, α Reg For L Reg The corresponding weight; L Des is the consistency loss of the descriptor, t is a hyperparameter used to control the penalty strength, the superscripts 1 and 2 of des represent the first and second images, i and k / k' represent the i-th descriptor of the first image and the k / k'th descriptor of the second image, the subscript o / o' of des represents that this is the o / o'th descriptor of the associated point set, n1 represents the total number of descriptor pairs, the distance between each pair of descriptors in the numerator is less than 8 pixels, corresponding to the i,k point pair, and the distance in the denominator is greater than 8, corresponding to the i,k' point pair, α Des For L Des The corresponding weight; L Dec To describe the correlation loss of the sub-element, the related_coefficient_sum function calculates the correlation coefficients between different dimensions and sums them up. Dec YesL Dec The corresponding weights; The image pair (o′1, o′2) is input into the constructed neural network model, and the corresponding feature points are obtained after inputting o′1, including its score Location And the descriptor The superscript 1 represents the first image o′1, and the subscript k represents the kth feature point; similarly, after inputting o′2, the corresponding Among them, the score and descriptor are the outputs of the corresponding grid positions, and the position of the feature point Then it is necessary to calculate the position output parameters of the network to obtain; The score, position, descriptor, and grid are specifically: After the image is input into the network, the corresponding outputs are obtained from the score output head, position output head and descriptor output head of the network. Assuming that the input image size is (h, w), the sizes of the score output head, position output head and descriptor output head are (1, h / 8, w / 8), (2, h / 8, w / 8), (256, h / 8, w / 8), respectively. The first one in the above brackets is the number of channels, and the last two are the height and width of the output result respectively. Assume that the size of the input image is 24*24, the output image is divided into 3*3 squares of size 8*8. Each square will have a score, two position parameters and a 256-dimensional descriptor, corresponding to the output of (1, h / 8, w / 8), (2, h / 8, w / 8), (256, h / 8, w / 8). The score of the feature point of the square is the output of the score output head corresponding to the square. Each grid has only one feature point, and its use will be determined based on the size of its score. The position of the square is determined by the two channels corresponding to the position output head. The parameters output by these two channels are in the range of 0-1, corresponding to the relative horizontal and vertical positions in the grid. The descriptor of the square is determined by the 256-dimensional vector of the corresponding descriptor branch. By analogy, the feature points of each square output, that is, its score, position and descriptor, can be obtained.
2. The robust visual feature matching and detection method for underground parking according to claim 1, characterized in that: The data to be tested is specifically a front view image of the vehicle taken by a monocular camera; the perspective transformation is specifically performed by calculating the homography matrix Apply this homography transformation to the front view image to obtain a bird's-eye view after perspective transformation. The homography matrix is Specifically: n=[0 0 -1] T Among them, K1 is the internal parameter of the non-bird's-eye view camera, and K2 is the internal parameter of the bird's-eye view virtual camera. is the camera external parameter, and d is the height of the virtual bird's-eye view camera.
3. The robust visual feature matching and detection method for underground parking according to claim 1, characterized in that: In step S3, the true value label is obtained by making a corner point. The specific process is: let the corner point be p i , the subscript i represents the i-th corner point, and is drawn on a graph of size (h / 4, w / 4) with p i / 4 is a local Gaussian distribution centered on the integer part, and the resulting graph is the true value label feature graph Y.
4. The robust visual feature matching and detection method for underground parking according to claim 2, characterized in that: The deep learning model for location detection in step S3 is specifically a neural network model based on the Center-net method: using the Hourglass network as the feature extraction layer, there are two output heads, one is a single-channel heat map, which is used to detect the area where the corner point may be located; the other is a dual-channel relative position map, which outputs the deviation to restore the precise position; The loss function of the location detection deep learning model is: Loss2=w1L seg +w2L off Among them, Loss2 is the total loss function of the deep learning model for location detection; L seg is the corner probability loss, Y is the true value label feature map, is a single-channel feature map output by the network. The subscripts x and y are the indices in the corresponding feature map. α and β are hyperparameters. xy =1, 1 noslots The value is 0, otherwise it is 1, N is the number of corner points, w1 is L seg The corresponding weight, w2 is L off The corresponding weight; L off is the deviation regression loss, the wavy line represents rounding down, O is the dual-channel feature map output by the network, p is the corner point, and k is the index of p and O.
5. The robust visual feature matching and detection method for underground parking according to claim 4, characterized in that: The specific process of obtaining the storage location detection result in step S4 is: screening according to the heat map output by the storage location detection neural network, selecting points on the map that are greater than the score threshold as candidate points, and then performing non-maximum suppression (NMS) to obtain storage location candidate points, and the result is: p out =4p seg +p off Among them, p seg is the position of the heat map where the score is greater than the threshold, p off It is the residual result of the corresponding dual-channel image output.
6. The robust visual feature matching and detection method for underground parking according to claim 5, characterized in that: The step S5 specifically comprises the following steps: S51, inversely transform the storage location corner points of the storage location detection result to the front view to generate the front view corner points. The specific formula of the inverse transformation is: in, is the corner point of the storage location, is the homography matrix, is the front view corner point; S52, according to the front view corner point The descriptor at the corner point of the front view on the feature map is obtained to complete the matching. The descriptor d at the corner point of the front view on the feature map is slot The coordinates are: in, and They are The x and y coordinates of the numpy array are rounded down. After obtaining the block where the front view corner point is located, we can get its corresponding descriptor, that is, d slot of x,y Using this descriptor for matching can well correlate the corner points of the warehouse locations between subsequent different frames.
