Deep learning-based pure RGB trolley semantic map construction method and deep learning-based pure RGB trolley semantic map relocation method
Through deep learning models, RGB images are used to estimate depth and detect key points, and a rich semantic map is built, which solves the problem that visual mapping cannot provide dense depth information, and achieves high-precision positioning and repositioning.
Patent Information
- Application Number
- CN202411067622.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-06
- Publication Date
- 2024-11-19
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing visual-based semantic maps cannot provide dense depth information, resulting in inaccurate positioning information and ineffective relocation.
Using the pure RGB car semantic map construction method based on deep learning, we collect pictures in real time through the RGB camera, use the deep learning model to perform RGB depth estimation, obtain the 3D coordinates of key points and the Rt matrix of the relative origin, build a rich semantic map, and then build a Position correction is performed through feature matching and optimization algorithms during relocation.
The dense depth information estimation of a monocular camera is realized, detailed scene depth information is provided, positioning accuracy and repositioning accuracy is improved, and the problem that visual mapping cannot provide dense depth information.
Smart Images

Figure CN118980368A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of intelligent vehicles, and in particular to a pure RGB vehicle semantic map construction method and a relocation method based on deep learning. Background Art
[0002] With the continuous development of science and technology, the level of automation and digitalization in the manufacturing industry is constantly improving. The emergence of smart cars is the key to realizing the intelligence of factories and warehouses. It is mainly used in smart warehousing, automated workshops, mechanical processing, home appliance production, tobacco, medical and food fields, significantly improving work efficiency and reducing labor costs. Positioning is very important during the operation of the car, and the positioning accuracy is strongly dependent on the accuracy of map construction, semantic information, etc.
[0003] The current mapping method is mainly based on LiDAR mapping. LiDAR-based mapping can provide depth information and has high accuracy. However, due to its less semantic information, positioning is mostly obtained by fusing odometer and point cloud matching results, resulting in existing positioning information being strongly dependent on prior information. When the odometer error is serious, positioning is often lost and repositioning is impossible. Other methods need to be introduced to assist positioning, and the price of LiDAR is relatively high. Some of the existing technologies are also based on vision. Compared with LiDAR mapping, only cameras are needed for image acquisition, which is low in cost. Cameras can provide rich visual information, including color, texture, shape, etc., so that vision-based mapping methods can better capture the details and features of the environment, and thus provide rich semantic information. However, vision-based mapping methods cannot provide dense depth information. The maps created are usually sparse and cannot provide a complete and detailed depth structure of the scene, which may cause the depth information in the map to be inconsistent with the depth of the real scene. Summary of the invention
[0004] Therefore, it is necessary to provide a pure RGB car semantic map construction method based on deep learning to solve the problem that the existing vision-based semantic map cannot provide dense depth information.
[0005] To achieve the above object, the present invention provides a method for constructing a pure RGB car semantic map based on deep learning, which comprises the following steps:
[0006] The car moves along a preset path in the scene. The RGB camera selects a certain posture to collect RGB images in real time and obtains the posture of the RGB camera and the posture of the car.
[0007] Use the deep learning model to perform RGB depth estimation on the RGB image to obtain a depth estimation map;
[0008] Perform key point detection and descriptor analysis on RGB images to obtain the 2D coordinates and descriptors of key points;
[0009] Obtain the depth of the key point through the 2D coordinates of the key point in the depth estimation map;
[0010] According to the camera internal parameters, the 2D coordinates of the key points and the depth of the key points, the 3D coordinates of the key points and the Rt matrix of the car relative to the origin in the corresponding RGB image are obtained;
[0011] The 2D coordinates of all key points in the corresponding RGB image, the 3D coordinates of the key points, the descriptors, and the Rt matrix of the car relative to the origin in the corresponding RGB image are associated and saved in the semantic map library.
[0012] Furthermore, the deep learning model is a YOLOv9 model or a model improved based on the YOLOv9 model.
[0013] Furthermore, the improved model based on the YOLOv9 model includes setting the activation function of the backbone network of the YOLOv9 model to a ReLU function.
[0014] Furthermore, the improved model based on the YOLOv9 model includes replacing each ELEN branch in the backbone network RepNCSPELEN4 module of the YOLOv9 model from Sequential to 4 groups of group convolutions.
[0015] Furthermore, the improved model based on the YOLOv9 model includes adding a labeled 3x3 convolution structure that does not change the feature size before each pooling layer of the SPP module in the neck of the YOLOv9 model.
[0016] Furthermore, the improved model based on the YOLOv9 model includes replacing RepNCSPELEN4 in the neck part of the YOLOv9 model with a standard ELEN.
[0017] Furthermore, the improved model based on the YOLOv9 model includes deleting the anchor-free structure in the head of the YOLOv9 model, first upsampling at the corresponding position, and then adding a convolution layer with 1 convolution layer and a convolution kernel size of 1x1 as the output segmentation head, and calculating the IOU loss of the three output heads respectively as the final loss.
[0018] Furthermore, the AKAZE algorithm is used to perform key point detection and descriptor analysis on the RGB image to obtain the 2D coordinates and descriptors of the key points.
[0019] The pure RGB car semantic map relocation method based on deep learning applies the above-mentioned pure RGB car semantic map construction method based on deep learning, which includes the following steps:
[0020] When the car is in motion, the RGB camera collects the current RGB image in real time;
[0021] Perform key point detection and descriptor calculation on the current RGB image to obtain the 2D coordinates and descriptors of the key points of the current RGB image;
[0022] The descriptor of the current RGB image is matched with the historical descriptor in the semantic map library and the similarity is calculated to obtain the most similar historical RGB image in the semantic map library;
[0023] Match the key points of the current RGB image with the most similar historical RGB image;
[0024] Estimate the Rt matrix of the matched key points to obtain the RT transformation matrix from the historical RGB image to the current RGB image;
[0025] Read the z-axis transformation according to the RT transformation matrix from the historical RGB image to the current RGB image; when z>0.1, the current RGB image matching result is incorrect and the current match is discarded; otherwise, multiply the RT transformation matrix from the historical RGB image to the current RGB image, the Rt matrix of the car relative to the origin in the historical RGB image, the historical RGB camera pose and the historical car pose position to obtain the current car pose.
[0026] Furthermore, the step of estimating the Rt matrix of the matched key points and the RT matrix of the historical RGB image to the current RGB image includes the following steps:
[0027] Read the 3D coordinates of the key points matched in the historical RGB image, and connect the 2D coordinates of the key points in the current RGB image with the 3D coordinates of the key points matched in the historical RGB image;
[0028] Use the RANSAC algorithm to optimize the feature matching results;
[0029] Use the optimized matching points to perform pose estimation using the PNP algorithm to obtain the RT transformation matrix from the historical RGB image to the current RGB image.
[0030] Different from the existing technology, the above technical solution mainly realizes the dense depth information estimation of the monocular camera through deep learning, and then obtains the depth information of the corresponding key points in the depth map according to the 2D coordinates of the key points, so as to calculate the 3D coordinates of the key points and the Rt matrix of the car relative to the origin. By associating the 2D coordinates, 3D coordinates, descriptors, the Rt matrix of the car relative to the origin, the pose of the RGB camera and the pose of the car of these key points, a rich semantic map can be constructed, in which each key point contains detailed information, which solves the problem that visual mapping cannot provide dense depth information and is helpful for subsequent scene understanding and decision-making tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 It is a flowchart of a method for constructing a pure RGB car semantic map based on deep learning in an embodiment;
[0032] Figure 2 It is a flowchart of the deep learning-based pure RGB car semantic map relocalization method in the embodiment. DETAILED DESCRIPTION
[0033] In order to explain the technical content, structural features, achieved objectives and effects of the technical solution in detail, the following is a detailed description in conjunction with specific embodiments and accompanying drawings.
[0034] Reference to "embodiment" herein means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The term "embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment, nor does it particularly limit its independence or association with other embodiments. In principle, in the present application, as long as there is no technical contradiction or conflict, the various technical features mentioned in the embodiments can be combined in any way to form a corresponding implementable technical solution.
[0035] Unless otherwise defined, the technical terms used in this document have the same meanings as those generally understood by those skilled in the art to which this application belongs; the use of relevant terms in this document is only for describing specific embodiments and is not intended to limit this application.
[0036] In the description of this application, the term "and / or" is an expression used to describe the logical relationship between objects, indicating that three relationships may exist, for example, A and / or B, which means: A exists, B exists, and A and B exist at the same time. In addition, the character " / " in this article generally indicates that the objects before and after are in an "or" logical relationship.
[0037] In the present application, terms such as “first” and “second” are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship of quantity, priority or sequence between these entities or operations.
[0038] Without further limitations, in this application, the words "include", "comprises", "has" or other similar expressions used in the sentences are intended to cover non-exclusive inclusion. These expressions do not exclude the presence of additional elements in the process, method or product including the elements, so that the process, method or product including a series of elements may include not only those limited elements, but also other elements not explicitly listed, or also include elements inherent to such process, method or product.
[0039] Similar to the understanding in the Patent Examination Guidelines, in this application, expressions such as "greater than", "less than", "exceed" and the like are understood to exclude the number itself; expressions such as "above", "below", "within" and the like are understood to include the number itself. In addition, in the description of the embodiments of this application, "multiple" means more than two (including two), and similar expressions related to "multiple" are also understood in this way, such as "multiple groups", "multiple times", etc., unless otherwise clearly and specifically limited.
[0040] In the description of the embodiments of the present application, space-related expressions used, such as "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "vertical", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc., indicate the orientation or position relationship based on the orientation or position relationship shown in the specific embodiments or drawings, and are only for the convenience of describing the specific embodiments of the present application or facilitating the reader's understanding, and do not indicate or imply that the referred device or component must have a specific position, a specific orientation, or be constructed or operated in a specific orientation, and therefore cannot be understood as a limitation on the embodiments of the present application.
[0041] Unless otherwise expressly specified or limited, in the description of the embodiments of the present application, the terms such as "install", "connect", "connect", "fix", "set", etc. used should be understood in a broad sense. For example, the "connection" can be a fixed connection, a detachable connection, or an integrated setting; it can be a mechanical connection, an electrical connection, or a communication connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. For technicians in the technical field to which the present application belongs, the specific meanings of the above terms in the embodiments of the present application can be understood according to the specific circumstances.
[0042] See also Figure 1 As shown, the present invention provides a pure RGB car semantic map construction method based on deep learning, and proposes a complete set of semantic map construction based on deep learning, which mainly realizes dense depth information estimation of a monocular camera through deep learning, and provides more detailed scene depth information; then the depth information of the corresponding key point is obtained in the depth map according to the 2D coordinates of the key point, so as to calculate the 3D coordinates of the key point, which can provide more accurate position information for each key point; then the 3D pose is calculated by the depth information of the key point and the 2D coordinates of the key point and the camera internal parameters to obtain the Rt matrix of the car relative to the origin; the 2D coordinates, 3D coordinates, descriptors, the Rt matrix of the car relative to the origin, the pose of the RGB camera and the pose of the car of these key points are associated, so as to construct a rich semantic map, in which each key point contains detailed information, which solves the problem that visual mapping cannot provide dense depth information, and is helpful for subsequent scene understanding and decision-making tasks.
[0043] To further illustrate the method for constructing a pure RGB car semantic map based on deep learning of the present invention, a specific implementation method is provided below, which includes:
[0044] The car moves along a preset path in the scene. The RGB camera selects a certain posture to collect RGB images in real time and obtains the posture of the RGB camera and the posture of the car.
[0045] Use the deep learning model to perform RGB depth estimation on the RGB image to obtain a depth estimation map;
[0046] Perform key point detection and descriptor analysis on RGB images to obtain the 2D coordinates and descriptors of the key points;
[0047] Obtain the depth of the key point through the 2D coordinates of the key point in the depth estimation map;
[0048] According to the camera internal parameters, the 2D coordinates of the key points and the depth of the key points, the 3D coordinates of the key points and the Rt matrix of the car relative to the origin in the corresponding RGB image are obtained;
[0049] The 2D coordinates of all key points in the corresponding RGB image, the 3D coordinates of the key points, the descriptors, the Rt matrix of the car relative to the origin in the RGB image, the pose of the RGB camera and the pose of the car are associated and saved in the semantic map library.
[0050] The above-mentioned car moves along a preset path in the scene. The car can be pushed along the preset path manually, or the car can be controlled manually to automatically move along the preset path.
[0051] The above-mentioned RGB camera selects an RGB image collected in real time at a certain pose. In order to improve the accuracy of key point detection and depth estimation, the collected RGB image can be preprocessed before key point detection and depth estimation, such as grayscale conversion, histogram equalization and other operations.
[0052] The above-mentioned deep learning model can be based on a convolutional neural network model (CNN), a YOLOE series model or other deep learning models, or even a model improved based on the above-mentioned models, and then trained with a large number of RGB images, so that the model can perform RGB depth estimation based on real-time RGB images to obtain a depth estimation map.
[0053] Taking the YOLOv9 model as an example, in order to enable the YOLOv9 model to realize the dense depth estimation function, and considering that the vehicle-side deployment has poor adaptability to some modules in YOLOv9, the YOLOv9 model is improved. The improved modules may be some modules and detection heads, etc. Specifically, a variety of embodiments are provided as follows.
[0054] In some embodiments, the improved model based on the YOLOv9 model includes setting the activation function of the backbone network of the YOLOv9 model to the ReLU function. Setting the activation function of the backbone network of the YOLOv9 model to the ReLU function can increase the expressive power of the model, accelerate the training speed, reduce the computational complexity, combat the vanishing gradient, etc., thereby improving the performance and effect of the model.
[0055] In some embodiments, the improved model based on the YOLOv9 model includes replacing each ELEN branch in the backbone network RepNCSPELEN4 module of the YOLOv9 model by Sequential with 4 groups of group convolutions (groupconvolution, g=4), and using 4 groups of group convolutions to reduce the number of parameters of each ELEN branch and reduce the complexity of the model.
[0056] In some embodiments, the improved model based on the YOLOv9 model includes adding a labeled 3x3 convolution structure that does not change the feature size before each pooling layer of the SPP module in the neck of the YOLOv9 model; in the SPP (SpatialPyramid Pooling) module, pooling layers of different sizes are used to capture features of different scales. However, due to the existence of the pooling operation, the spatial information of the feature is lost. By adding a 3x3 convolution structure that does not change the feature size before each pooling layer, the feature loss caused by SPP can be supplemented to maintain the integrity of the feature.
[0057] In some embodiments, the improved model based on the YOLOv9 model includes replacing the RepNCSPELEN4 of the neck part of the YOLOv9 model with the standard ELEN, and the auxiliary branch part remains unchanged. RepNCSPELEN4 with the standard ELEN can reduce the number of parameters of the model; compared with the RepNCSPELEN4 structure, the ELEN structure reduces some convolutional layers and parameters in each branch, thereby reducing the complexity of the model.
[0058] In some embodiments, the improved model based on the YOLOv9 model includes deleting the anchor-free structure in the head of the YOLOv9 model, first upsampling at the corresponding position, and then adding a convolution layer with 1 convolution layer and a convolution kernel size of 1x1 as the output segmentation head, and calculating the IOU loss for the three output heads respectively and using it as the final loss; by adding the segmentation head to the model, the target can be segmented at the pixel level to obtain a finer target boundary, improve the accuracy of target detection and the precision of target segmentation, and realize multi-task learning. By calculating the IOU loss as the final loss, the results of target detection and segmentation can be more accurately evaluated, which helps to optimize the model parameters and improve the training effect and generalization ability of the model.
[0059] Of course, the above-mentioned multiple embodiments can also be combined to achieve a combination of multiple beneficial effects, so that the model improved based on the YOLOv9 model can achieve better performance.
[0060] In the above, key point detection and descriptor analysis are performed on the RGB image to obtain the 2D coordinates and descriptors of the key points. The key points can be corner points, edges or other points with significant properties in the image, and the descriptors are feature descriptions of the image area around the key points. Key point detection and descriptor analysis are performed on the RGB image to obtain the 2D coordinates and descriptors of the key points. SIFT, SURF (accelerated robust features), ORB (direction tilt fast algorithm), AKAZE (accelerated KAZE), BRISK (accelerated rotation invariant features) and other algorithms can be used to perform key point detection and descriptor analysis on the RGB image to obtain the 2D coordinates and descriptors of the key points. The AKAZE algorithm is preferably used. The AKAZE algorithm is an improvement on the KAZE algorithm. While maintaining the advantages of the KAZE algorithm, the calculation efficiency is improved by using acceleration technology. The AKAZE algorithm has good invariance to scale changes, affine transformations and viewpoint changes.
[0061] The above obtains the 3D coordinates of the key point and the Rt matrix of the car relative to the origin in the corresponding RGB image based on the camera intrinsic parameters, the 2D coordinates of the key point and the depth of the key point.
[0062] Assume that the camera intrinsic matrix Where fx and fy are the camera focal lengths, and cx and cy are the camera principal points.
[0063] The 2D coordinates (u, v) of the key point represent the pixel coordinates of the key point in the RGB image.
[0064] The depth d of the keypoint represents the depth of the keypoint relative to the camera.
[0065] The calculation of the 3D coordinates of the key points includes the following steps:
[0066] First, the camera intrinsic parameter matrix K is used to convert the pixel coordinates (u, v) to normalized coordinates (x, y, 1), where x = (u-cx) / fx, y = (v-cy) / fx.
[0067] Then, calculate the 3D coordinates (X, Y, Z) of the key points:
[0068] X=x*d
[0069] Y=y*d
[0070] Z=d.
[0071] The calculation of the Rt matrix of the car relative to the origin in the RGB image includes the following steps:
[0072] For the Rt matrix of the car relative to the origin, the 3D coordinates (X, Y, Z) of the key points need to be converted to the camera coordinate system, and the Rt matrix of the camera relative to the origin is calculated using the rotation matrix R and the translation vector t.
[0073] The rotation matrix R can be obtained from the camera's Euler angles (yaw, pitch, roll) or quaternion representation. The translation vector t can be obtained from the camera pose coordinates. Of course, you can also choose a PnP algorithm (such as RANSAC-PnP) to estimate the camera's rotation matrix R and translation vector t. Of course, if a more accurate pose estimate is required, an optimization algorithm (such as Levenberg-Marquardt) can be used to iteratively optimize the initial estimated pose to minimize the reprojection error.
[0074] The present invention also provides a pure RGB car semantic map repositioning method based on deep learning, which applies the above-mentioned pure RGB car semantic map construction method based on deep learning to determine or correct the position and posture of the car based on the constructed semantic map. When the car needs to be positioned in a new environment, it can be matched with the existing semantic map to find the best estimated value of the position and posture. This can be achieved through feature matching, geometric constraints, optimization algorithms and other methods.
[0075] See also Figure 2As shown, the present invention uses a feature matching method as an embodiment to illustrate a pure RGB car semantic map relocation method based on deep learning, which includes the following steps:
[0076] When the car is in motion, the RGB camera collects the current RGB image in real time;
[0077] Perform key point detection and descriptor calculation on the current RGB image to obtain the 2D coordinates and descriptors of the key points of the current RGB image;
[0078] The descriptor of the current RGB image is matched with the historical descriptor in the semantic map library and the similarity is calculated to obtain the most similar historical RGB image in the semantic map library;
[0079] Match the key points of the current RGB image with the most similar historical RGB image;
[0080] Estimate the Rt matrix of the matched key points to obtain the RT transformation matrix from the historical RGB image to the current RGB image;
[0081] Read the z-axis transformation according to the RT transformation matrix from the historical RGB image to the current RGB image; when z>0.1, the matching result of the current RGB image is incorrect and the current match is discarded; otherwise, multiply the RT transformation matrix from the historical RGB image to the current RGB image, the Rt matrix of the car relative to the origin in the historical RGB image, the pose of the historical RGB camera and the pose of the historical car to obtain the current pose of the car.
[0082] The above historical RGB images, historical descriptors, the Rt matrix of the car relative to the origin in the historical RGB images, the pose of the historical RGB camera and the pose of the historical car are the RGB images and descriptors saved in the semantic map library when constructing the pure RGB car semantic map based on deep learning, as well as the Rt matrix of the car relative to the origin in the corresponding RGB images, the pose of the RGB camera and the pose of the car.
[0083] The above step of estimating the Rt matrix of the matched key points and the RT matrix of the historical RGB image to the current RGB image includes the following steps:
[0084] Read the 3D coordinates of the key points matched in the historical RGB image, and connect the 2D coordinates of the key points in the current RGB image with the 3D coordinates of the key points matched in the historical RGB image;
[0085] Use the RANSAC algorithm to optimize the feature matching results;
[0086] Use the optimized matching points to perform pose estimation using the PNP algorithm to obtain the RT transformation matrix from the historical RGB image to the current RGB image.
[0087] It should be noted that, although the above embodiments have been described in this article, the patent protection scope of the present invention is not limited thereby. Therefore, based on the innovative concept of the present invention, changes and modifications made to the embodiments described herein, or equivalent structures or equivalent process changes made using the contents of the present invention specification and drawings, directly or indirectly applying the above technical solutions to other related technical fields, are all included in the patent protection scope of the present invention.
Claims
1. A method for constructing pure RGB car semantic maps based on deep learning. It is characterized in that The following steps are involved: The car moves along a preset path in the scene. The RGB camera selects a certain posture to collect RGB images in real time and obtains the posture of the RGB camera and the posture of the car. Use the deep learning model to perform RGB depth estimation on the RGB image to obtain a depth estimation map; Perform key point detection and descriptor analysis on RGB images to obtain the 2D coordinates and descriptors of key points; Obtain the depth of the key point through the 2D coordinates of the key point in the depth estimation map; According to the camera internal parameters, the 2D coordinates of the key points and the depth of the key points, the 3D coordinates of the key points and the Rt matrix of the car relative to the origin in the corresponding RGB image are obtained; The 2D coordinates of all key points in the corresponding RGB image, the 3D coordinates of the key points, the descriptors, the Rt matrix of the car relative to the origin in the RGB image, the pose of the RGB camera and the pose of the car are associated and saved in the semantic map library.
2. According to the method for constructing a pure RGB car semantic map based on deep learning according to claim 1, It is characterized in that The deep learning model is a YOLOv9 model or a model improved based on the YOLOv9 model.
3. According to the method for constructing a pure RGB car semantic map based on deep learning according to claim 2, It is characterized in that The improved model based on the YOLOv9 model includes setting the activation function of the backbone network of the YOLOv9 model to the ReLU function.
4. According to the method for constructing a pure RGB car semantic map based on deep learning according to claim 2, It is characterized in that The improved model based on the YOLOv9 model includes replacing each ELEN branch in the backbone network RepNCSPELEN4 module of the YOLOv9 model from Sequential to 4 groups of group convolutions.
5. According to the method for constructing a pure RGB car semantic map based on deep learning according to claim 2, It is characterized in that The improved model based on the YOLOv9 model includes adding a labeled 3x3 convolution structure that does not change the feature size before each pooling layer of the SPP module in the neck of the YOLOv9 model.
6. The method for constructing a pure RGB car semantic map based on deep learning according to claim 2, It is characterized in that The improved model based on the YOLOv9 model includes replacing RepNCSPELEN4 in the neck part of the YOLOv9 model with the standard ELEN.
7. The method for constructing a pure RGB car semantic map based on deep learning according to claim 2, It is characterized in that The improved model based on the YOLOv9 model includes deleting the anchor-free structure in the head of the YOLOv9 model, first upsampling at the corresponding position, and then adding a convolution layer with 1 convolution layer and a convolution kernel size of 1x1 as the output segmentation head, and calculating the IOU loss of the three output heads respectively as the final loss.
8. The method for constructing a pure RGB car semantic map based on deep learning according to claim 1, It is characterized in that Use the AKAZE algorithm to perform key point detection and descriptor analysis on RGB images to obtain the 2D coordinates and descriptors of the key points.
9. A pure RGB car semantic map relocation method based on deep learning, applying the pure RGB car semantic map construction method based on deep learning described in any of claims 1 to 8, It is characterized in that The following steps are involved: When the car is in motion, the RGB camera collects the current RGB image in real time; Perform key point detection and descriptor calculation on the current RGB image to obtain the 2D coordinates and descriptors of the key points of the current RGB image; The descriptor of the current RGB image is matched with the historical descriptor in the semantic map library and the similarity is calculated to obtain the most similar historical RGB image in the semantic map library; Match the key points of the current RGB image with the most similar historical RGB image; Estimate the Rt matrix of the matched key points to obtain the RT transformation matrix from the historical RGB image to the current RGB image; Read the z-axis transformation according to the RT transformation matrix from the historical RGB image to the current RGB image; When z>0.1, the current RGB image matching result is incorrect and the current match is eliminated; Otherwise, the RT transformation matrix from the historical RGB image to the current RGB image, the Rt matrix relative to the origin in the historical RGB image, the pose of the historical RGB camera, and the pose position of the historical car are multiplied to obtain the current pose of the car.
10. The pure RGB car semantic map relocation method based on deep learning according to claim 9, It is characterized in that The step of estimating the Rt matrix of the matched key points and the RT matrix of the historical RGB image to the current RGB image includes the following steps: Read the 3D coordinates of the key points matched in the historical RGB image, and connect the 2D coordinates of the key points in the current RGB image with the 3D coordinates of the key points matched in the historical RGB image; Use the RANSAC algorithm to optimize the feature matching results; Use the optimized matching points to perform pose estimation using the PNP algorithm to obtain the RT transformation matrix from the historical RGB image to the current RGB image.
