Lightweight self-supervision visual mileage calculation method of multi-feature fusion architecture and application of lightweight self-supervision visual mileage calculation method
By using a lightweight self-supervised visual odometry architecture with a multi-feature fusion architecture, and by fusing deep semantic prediction networks and pose prediction networks with depth and semantic features, this approach solves the problems of complex model architecture and limited feature representation in existing technologies, and achieves high-precision and high-speed camera pose estimation.
Patent Information
- Application Number
- CN202411309263.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-19
- Publication Date
- 2025-12-12
AI Technical Summary
Existing self-supervised visual odometry models are difficult to achieve high-precision and high-speed camera pose estimation in real-world applications due to their complex model architecture and limited feature representation.
A multi-feature fusion architecture is adopted, which extracts deep features and semantic features through a deep semantic prediction network and a pose prediction network, and then fuses them to obtain pose features. This reduces the need for a backbone network structure for feature extraction and enhances the feature representation capability of pose estimation by utilizing a spatial attention mechanism.
While ensuring the algorithm's running speed, the accuracy of pose estimation has been significantly improved, meeting the accuracy and robustness requirements of practical application scenarios.
Smart Images
Figure CN121120735A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of visual odometry, and particularly relates to a lightweight self-supervised visual odometry algorithm with a multi-feature fusion architecture and application thereof. BACKGROUND
[0002] As an important part of visual SLAM (simultaneous localization and mapping) system, the visual odometry algorithm mainly uses image sequences obtained by a camera to calculate the camera pose, so as to estimate the motion trajectory, and can be widely applied to the fields of autonomous robots, augmented reality, virtual reality and the like. The existing visual odometry algorithm can be mainly divided into two categories: a geometric-based visual odometry algorithm and a deep learning-based visual odometry algorithm. The former has the key problems of poor robustness to environmental conditions and poor feature representation capability, and it is difficult to achieve satisfactory camera pose estimation performance in most application scenarios. The latter realizes feature extraction through a deep neural network, and uses deep feature information to ensure the accuracy and robustness of the pose estimation of the visual odometry algorithm in complex scenes.
[0003] The deep learning-based visual odometry algorithm can be divided into supervised training and self-supervised training, the former needs a large number of true value labels provided by a data set, and brings the problem of high cost in some application backgrounds. The latter introduces depth estimation and pose estimation results to construct a self-supervised signal through the photometric consistency assumption, and then the model does not need to provide any label in the training process, and has become the mainstream research and application direction.
[0004] At present, the existing self-supervised training visual odometry has the problems of complex model architecture and limited feature representation, and thus it is difficult to achieve high-precision and high-speed camera pose estimation in actual application scenarios. The specific reasons are as follows: first, some existing visual odometry algorithms extract depth semantic features and pose features through two independent feature extraction backbone networks, resulting in high algorithm complexity and operation amount, and slow running speed; second, some existing algorithms attempt to extract depth semantic features and pose features through a shared feature extraction backbone network, but do not fully consider the difference between the pose features and the depth semantic features, resulting in difficulty in extracting sufficient information for completing the task of the visual odometry, and limited accuracy and robustness; third, the existing algorithms ignore the importance of feature sharing and fusion between tasks, resulting in difficulty in meeting the actual demand for algorithm accuracy. SUMMARY
[0005] The application aims to provide a lightweight self-supervised visual odometry algorithm with a multi-feature fusion architecture and application thereof, so as to solve the technical problem that the existing self-supervised training visual odometry has complex model architecture and limited feature representation, and thus it is difficult to achieve high-precision and high-speed camera pose estimation in actual application scenarios.
[0006] To achieve the above objectives, the first aspect of this application provides a lightweight self-supervised visual odometry algorithm with a multi-feature fusion architecture, comprising:
[0007] Acquire the target frame image and its neighboring frame images, and input them into a pre-trained deep learning model;
[0008] Based on the output of the deep learning model, the depth feature map, semantic feature map, and pose change matrix of the target frame image are obtained.
[0009] The deep learning model includes a deep semantic prediction network and a pose prediction network. The deep semantic prediction network is used to concatenate the target frame image and the adjacent frame images and extract features to obtain a first feature map. Depth features and semantic features are extracted from the first feature map, and the depth feature map and the semantic feature map are output respectively.
[0010] The pose prediction network is used to fuse the depth feature map and the semantic feature map and extract pose features to obtain spatial pose features; extract features from the spatial pose features and output the pose change matrix.
[0011] In one or more embodiments, the step of extracting depth features from the first feature map includes:
[0012] The first feature map is subjected to convolution and upsampling operations in sequence to obtain a resolution equal to that of the first feature map. Figure One The first depth key feature map;
[0013] The first feature map and the first depth key feature map are concatenated, and then a convolution operation is performed to obtain the first depth fusion feature map.
[0014] The first deep fusion feature map is upsampled and then activated by a nonlinear activation function to obtain the deep feature map.
[0015] In one or more embodiments, the step of extracting semantic features from the first feature map includes:
[0016] The first feature map is convolved, activated by a non-linear activation function, and then upsampled to obtain a resolution equal to that of the first feature map. Figure One The first semantic key feature map;
[0017] The first semantic key feature map and the first feature map are added together to obtain the first semantic fusion feature map;
[0018] The first semantic fusion feature map is subjected to a convolution operation, and then is activated by a nonlinear activation function to obtain the semantic feature map.
[0019] In one or more embodiments, the step of fusing the depth feature map and the semantic feature map and extracting a pose feature comprises:
[0020] The depth feature map and the semantic feature map are spliced, and then subjected to a max-pooling operation and an average-pooling operation respectively, and the two operation results are spliced again to obtain a second fusion feature map;
[0021] The second fusion feature map is subjected to a convolution operation, and then is activated by a nonlinear activation function to obtain the second feature map;
[0022] The second feature map is subjected to a convolution operation and a batch normalization operation in sequence, and is activated by a nonlinear activation function to obtain a spatial enhancement feature map;
[0023] The second feature map and the spatial enhancement feature map are fused to obtain a spatial pose feature.
[0024] In one or more embodiments, the step of extracting a feature from the spatial pose feature and outputting the pose change matrix comprises:
[0025] The spatial pose feature is subjected to several convolution operations, and is activated by a nonlinear activation function after each convolution operation, and then is subjected to a convolution operation, a matrix average operation and a matrix dimension change operation in sequence to obtain a pose change matrix.
[0026] In one or more embodiments, the step of extracting a feature from the spatial pose feature and outputting the pose change matrix comprises:
[0027] F c1 =ReLU(Conv1(Ff out ))
[0028] F c3 =ReLU(Conv3(ReLU(Conv3(F c1 ))))
[0029] F p =Conv3(F c3 )
[0030] R,t=reshape(mean(0.01×F p ))
[0031] In the formula, F fout For the spatial pose features, ReLU is a non-linear activation function, mean(·) is a matrix averaging operation, reshape(·) is a matrix dimension change operation, R,t is a rotation and translation matrix, i.e., a pose change matrix, Conv1 is a 1*1 convolution operation, and Conv3 is a 3*3 convolution operation.
[0032] In one or more embodiments, the training method of the deep learning model includes:
[0033] Based on the depth feature map and pose change matrix of the target frame image output by the deep learning model, a loss function is constructed.
[0034] Based on the loss function, the parameters of the deep neural network are updated along the direction of gradient descent until the loss function converges;
[0035] The loss function is composed of a weighted sum of the photometric loss between the reconstructed image and the target frame image, and the edge-aware depth smoothness loss of the depth feature map. The reconstructed image is constructed based on the depth feature map of the target frame image and the pose change matrix.
[0036] In one or more embodiments, the formula for calculating the photometric loss is as follows:
[0037]
[0038] In the formula, I t For the target frame image, I' t For the reconstructed image, I s For the adjacent frame images, p is the index value of the pixel coordinates, α is a hyperparameter, and L... ssim For structural similarity; and / or,
[0039] The formula for calculating the edge-aware depth smoothness loss is as follows:
[0040]
[0041] In the formula, It is the mean-normalized inverse depth map of the depth feature map, where i,j represent the pixel index values of the target frame image. and These represent the gradients in the x and y directions of the target frame image, respectively.
[0042] To achieve the above objectives, a second aspect of this application provides a lightweight self-supervised visual odometry architecture with a multi-feature fusion structure, comprising:
[0043] The input module is used to acquire the target frame image and its neighboring frame images and input them into the pre-trained deep learning model.
[0044] The output module is used to obtain the depth feature map, semantic feature map, and pose transformation matrix of the target frame image based on the output of the deep learning model.
[0045] The deep learning model includes a deep semantic prediction network and a pose prediction network. The deep semantic prediction network is used to concatenate the target frame image and the adjacent frame images and extract features to obtain a first feature map. Depth features and semantic features are extracted from the first feature map, and the depth feature map and the semantic feature map are output respectively.
[0046] The pose prediction network is used to fuse the depth feature map and the semantic feature map and extract pose features to obtain a second feature map; features are extracted from the second feature map and the pose change matrix is output.
[0047] To achieve the above objectives, a third aspect of this application provides an electronic device, comprising:
[0048] At least one processor; and
[0049] A memory that stores instructions that, when executed by the at least one processor, cause the at least one processor to execute a lightweight self-supervised visual odometry algorithm with a multi-feature fusion architecture as described in any of the above embodiments.
[0050] To achieve the above objectives, a fourth aspect of this application provides a machine-readable storage medium storing executable instructions that, when executed, cause the machine to perform a lightweight self-supervised visual odometry based on a multi-feature fusion architecture as described in any of the above embodiments.
[0051] The advantages of this application, which differ from existing technologies, are:
[0052] This application leverages spatial attention mechanism to efficiently fuse depth and semantic features, enhancing the feature representation capability of pose estimation and significantly improving the pose estimation accuracy of the algorithm in practical application scenarios while ensuring the algorithm's running speed.
[0053] In this application, the pose feature is extracted based on the fusion of deep features and semantic features. Only one feature extraction backbone network structure needs to be set, which greatly reduces the requirement for the feature extraction backbone network structure and improves the feature representation capability of the feature extraction backbone network, thus ensuring the accuracy and running speed of the algorithm in practical application scenarios. Attached Figure Description
[0054] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0055] Figure 1 This is a flowchart illustrating one implementation of the lightweight self-supervised visual odometry algorithm based on the multi-feature fusion architecture of this application.
[0056] Figure 2 This is a schematic diagram of the structure of one embodiment of the deep semantic prediction network of this application;
[0057] Figure 3 This is a flowchart illustrating one implementation method of the depth feature map acquisition method in this application;
[0058] Figure 4 This is a flowchart illustrating one implementation method of the semantic feature map acquisition method in this application;
[0059] Figure 5 This is a schematic diagram of the structure of one embodiment of the pose prediction network of this application;
[0060] Figure 6 This is a flowchart illustrating one implementation method of the spatial pose feature acquisition method of this application;
[0061] Figure 7 This is a flowchart illustrating one implementation method of the deep learning model training method of this application;
[0062] Figure 8 This is a schematic diagram of one embodiment of the lightweight self-supervised visual odometry with multi-feature fusion architecture of this application;
[0063] Figure 9 This is a schematic diagram of one embodiment of the electronic device of this application. Detailed Implementation
[0064] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of this application.
[0065] Currently, existing self-supervised visual odometry systems struggle to achieve high-precision, high-speed camera pose estimation in real-world applications. The specific reasons are as follows:
[0066] First, some existing semantic visual odometry algorithms based on self-supervised training extract deep semantic features and pose features separately through two independent feature extraction backbone networks. Since the feature extraction backbone network has high algorithm complexity and computational load in visual odometry, this redundant feature extraction backbone network architecture makes semantic visual odometry have significant problems in terms of lightweight model complexity, making it difficult to meet the practical application requirements of high running speed.
[0067] Second, some self-supervised training semantic visual odometry methods attempt to introduce a shared feature extraction backbone network to uniformly extract depth, semantic and pose features. However, these methods do not fully consider the differences between pose features and the other two features, making it difficult for the shared feature extraction backbone network to extract enough information to complete the semantic visual odometry task. As a result, the accuracy and robustness in practical applications are limited to a certain extent.
[0068] Third, existing semantic visual odometry methods based on self-supervised training neglect the importance of feature sharing and fusion between tasks, resulting in algorithm accuracy that is difficult to meet practical needs.
[0069] To address the aforementioned issues, the applicant developed a novel self-supervised visual odometry algorithm. This algorithm extracts pose features by fusing deep and semantic features, reducing the need for a backbone network structure for feature extraction and accelerating the algorithm's accuracy and speed.
[0070] Specifically, please refer to Figure 1 , Figure 1 This is a flowchart illustrating one implementation of the lightweight self-supervised visual odometry algorithm based on the multi-feature fusion architecture of this application.
[0071] like Figure 1 As shown, the algorithm includes:
[0072] S100: Obtain the target frame image and its adjacent frame images, and input them into the pre-trained deep learning model.
[0073] In this context, adjacent frame images are images that are continuous with the target frame image. For example, when the target frame image is frame t, the adjacent frame image can be frame t-1.
[0074] S200. Based on the output of the deep learning model, obtain the depth feature map, semantic feature map, and pose change matrix of the target frame image.
[0075] Based on the target frame image and adjacent frame images, the deep learning model can extract deep features and semantic features respectively, and output deep feature maps and semantic feature maps. At the same time, based on the fusion of deep feature maps and semantic feature maps, pose features can be extracted, the camera pose can be estimated when the target frame image is displayed, and thus the pose change matrix can be output.
[0076] Specifically, the deep learning model in this application includes a deep semantic prediction network and a pose prediction network. The deep semantic prediction network is used to concatenate the target frame image and adjacent frame images and extract features to obtain a first feature map. Depth features and semantic features are extracted from the first feature map, and the depth feature map and semantic feature map are output respectively.
[0077] The pose prediction network is used to fuse the depth feature map and the semantic feature map and extract pose features to obtain a second feature map; features are extracted from the second feature map and the pose change matrix is output.
[0078] The following section provides a detailed introduction to deep semantic prediction networks. Please refer to [link / reference]. Figure 2 , Figure 2 This is a schematic diagram of one embodiment of the deep semantic prediction network of this application.
[0079] like Figure 2 As shown, the deep semantic prediction network includes an image stitching module, which is used to stitch the target frame image and adjacent frame images together to form an image rich in spatiotemporal information.
[0080] The deep semantic prediction network includes a shared feature extraction backbone network, which is used to extract features from the fused image to obtain a first feature map.
[0081] Specifically, in this embodiment, the shared feature extraction backbone network can be any backbone network architecture commonly used in the field, such as ResNet network, MobileNet network, Vision Transformer network, etc., all of which can achieve the effect of this embodiment.
[0082] It should be noted that, in order to ensure the compatibility of image size and the extraction backbone network, the target frame image and adjacent frame images can be unified in resolution before image stitching. For example, the image resolution can be unified to 192*640. Then, the images are stitched together and input into the shared feature extraction backbone network to obtain the first feature map.
[0083] The deep semantic prediction network also includes a depth estimation network, which is used to extract depth features from the first feature map and output a depth feature map.
[0084] Specifically, please refer to Figure 3 , Figure 3This is a flowchart illustrating one implementation method of the depth feature map acquisition method in this application.
[0085] like Figure 3 As shown, methods for outputting depth feature maps by depth estimation networks include:
[0086] S301. Perform convolution and upsampling operations on the first feature map sequentially to obtain the resolution and the first feature map. Figure One The first depth key feature map.
[0087] First, key features can be extracted from the first feature map through convolution operations. Since the convolution operation reduces the spatial resolution of the feature map, the spatial dimension of the feature map can be enlarged to a size that matches the original input first feature map through upsampling operations, thereby restoring the integrity of spatial information and obtaining the first depth key feature map.
[0088] In one implementation, the specific operating formula can be as follows:
[0089] F upi =Upsample(Conv3(F x ))
[0090] In the formula, F x The first feature map is represented by Upsample, which is the upsampling operation, and Conv3 represents the convolution operation with a 3×3 kernel. upi This is the first depth key feature map.
[0091] S302. The first feature map and the first depth key feature map are concatenated, and then a convolution operation is performed to obtain the first depth fusion feature map.
[0092] To maximize the utilization of feature information, the detailed information of the first feature map of the original input is concatenated with the rich contextual information of the first deep key feature map after upsampling.
[0093] Then, convolution operations are applied again to further extract and fuse the features, thereby enhancing the feature representation capability and obtaining the first deep fusion feature map.
[0094] In one implementation, the specific operating formula can be as follows:
[0095] F upti =Conv3(Cat(F upi ,F xi ))
[0096] In the formula, Cat represents the matrix concatenation operation, and F upti This is the first deep fusion feature map.
[0097] S303. Upsample the first deep fusion feature map and then activate it using a non-linear activation function to obtain a deep feature map.
[0098] Furthermore, the first deep fusion feature map can be amplified by upsampling to restore its spatial resolution. Then, in order to introduce nonlinear factors to further enhance the feature representation capability, these amplified feature maps will be normalized by a nonlinear activation function to obtain the deep feature map.
[0099] In one implementation, the specific operating formula can be as follows:
[0100] F douti =Sigmoid(Upsample(Fupti)),
[0101] Among them, F douti For the depth feature map, Sigmid represents a non-linear activation function. In other embodiments, other non-linear activation functions can also be used, all of which can achieve the effect of this embodiment.
[0102] Please continue reading. Figure 2 The deep semantic prediction network also includes a semantic segmentation network, which is used to extract semantic features from the first feature map and output a semantic feature map.
[0103] Specifically, please refer to Figure 4 , Figure 4 This is a flowchart illustrating one implementation method of the semantic feature map acquisition method in this application.
[0104] like Figure 4 As shown, methods for semantic segmentation networks to output semantic feature maps include:
[0105] S401. Perform a convolution operation on the first feature map, activate it using a non-linear activation function, and then perform an upsampling operation to obtain the resolution and the first feature map. Figure One The first semantic key feature map of the origin.
[0106] First, the semantic segmentation network extracts key features from the first feature map of the input through convolution operations, and then normalizes and nonlinearly transforms these features through a nonlinear activation function.
[0107] Since the convolution operation reduces the spatial resolution of the feature map, in order to recover this lost spatial information, the spatial size of the feature map is enlarged to match the first feature map of the original input through an upsampling operation, thus obtaining the first semantic key feature map.
[0108] S402. Perform a feature addition operation on the first semantic key feature map and the first feature map to obtain the first semantic fusion feature map.
[0109] In order to effectively fuse information at different scales and levels, the magnified first semantic key feature map and the original first feature map are combined to obtain the first semantic fusion feature map.
[0110] In one implementation, the specific operating formulas for S401 and S402 can be as follows:
[0111] F ce =ELU(Conv3(F x ))
[0112] F add =Add(Upsample(F ce ),F x )
[0113] In the formula, F x The first feature map is defined by Upsample, ELU is a non-linear activation function, Add is a feature addition operation, and F... ce For the first semantic key feature map, F add This is the first semantic fusion feature map.
[0114] In other implementations, other types of nonlinear activation functions can also be used to achieve the same effect as this implementation.
[0115] S403. Perform a convolution operation on the first semantic fusion feature map, and then activate it through a non-linear activation function to obtain a semantic feature map.
[0116] To further refine the feature extraction and fusion process, convolution operations are applied again to process the feature map after feature addition, and nonlinear activation functions are used to normalize and transform the features to obtain the initial semantic feature map.
[0117] In one implementation, the specific operating formula can be as follows:
[0118] F sout =ELU(Conv3(F add )
[0119] In the formula, F sout This is a semantic feature map.
[0120] Based on the above Figure 3 and Figure 4 The steps shown obtain a depth feature map and a semantic feature map by extracting features from the first feature map. This enhances the algorithm's ability to represent and understand scene spatial features, thereby ensuring the depth estimation accuracy of the algorithm in dynamic scenes.
[0121] The pose prediction network is described in detail below. Please refer to [link / reference]. Figure 5 , Figure 5 This is a schematic diagram of one embodiment of the pose prediction network of this application.
[0122] like Figure 5 As shown, the pose prediction network includes a pose estimation enhancement network. The pose estimation enhancement network is used to extract and fuse semantic and deep features based on deep feature maps and semantic feature maps through an attention mechanism, and to extract pose features to obtain spatial pose features.
[0123] Specifically, please refer to Figure 6 , Figure 6 This is a flowchart illustrating one implementation method of the spatial pose feature acquisition method of this application.
[0124] like Figure 6 As shown, methods for enhancing the output spatial pose features of pose estimation networks include:
[0125] S601. Concatenate the depth feature map and the semantic feature map, then perform max pooling and average pooling operations respectively, and concatenate the results of the two operations again to obtain the second fused feature map.
[0126] First, concatenating the deep feature map and the semantic feature map enables the integration of deep features and semantic features, thereby fusing feature information from different levels.
[0127] Subsequently, in order to extract richer feature representations from the stitched feature map, max pooling and average pooling operations were used respectively. The two pooling operations focus on extracting local maxima and averages in the feature map, thereby capturing different feature patterns.
[0128] Finally, the operation structures of the two pooling operations are spliced together again to combine the different features extracted by the two.
[0129] In one implementation, the calculation formula for the above steps can be as follows:
[0130] F cat =Cat(F dout ,F sout )
[0131] F pool =Cat(MaxPool(F cat ),AvgPool(F cat ))
[0132] In the formula, MaxPool and AvgPool represent the max pooling operation and the average pooling operation, respectively, and F dout and F soutThese are depth feature maps and semantic feature maps, respectively. Cat represents the matrix concatenation operation, and F... pool This is the second fusion feature map.
[0133] In this embodiment, sigmoid is used as the non-linear activation function. In other embodiments, other non-linear activation functions can also be used to achieve the same effect.
[0134] S602. Perform a convolution operation on the second fused feature map, and then activate it through a non-linear activation function to obtain the second feature map.
[0135] To further extract and transform features from the concatenated feature map, a convolution operation is first applied, followed by normalization and nonlinear transformation of the convolutional features using a nonlinear activation function to obtain the second feature map.
[0136] In one implementation, the calculation formula for this step can be as follows:
[0137] F sam =Sigmoid(Conv3(F pool ))
[0138] In the formula, F sam This is the second feature map.
[0139] S603. Perform convolution and batch normalization operations on the second feature map in sequence, and activate it through a non-linear activation function to obtain a spatially enhanced feature map.
[0140] Furthermore, to enhance the accuracy of the pose estimation network from a spatial perspective, firstly, convolutional operations are used to extract key features from the input second feature map; to improve the stability of the feature map, the extracted feature map is normalized through a batch normalization operation; the normalized feature map is then passed through a nonlinear activation function to introduce nonlinear factors and enhance the representational power of the features, thereby obtaining a spatially enhanced feature map.
[0141] S604. The second feature map and the spatial augmentation feature map are fused to obtain the spatial pose feature.
[0142] To integrate the detailed information of the second feature map from the original input with the feature map after convolution and normalization, the spatial augmentation feature map is added element-wise to the second feature map to obtain the spatial pose feature.
[0143] In one implementation, the calculation formulas for S603 and S604 can be as follows:
[0144] F fout =ELU(Conv1BN(Fsam ))+F sam
[0145] In the formula, F sam For the second feature map, BN is the batch normalization operation, F fout This refers to spatial pose characteristics.
[0146] Please continue reading. Figure 5 The pose prediction network also includes a pose estimation network, which is used to decode the spatial pose features obtained above and output the final pose change matrix.
[0147] Specifically, methods for outputting pose change matrices based on spatial pose features include:
[0148] The spatial pose features are subjected to several convolution operations, and after each convolution operation, they are activated by a nonlinear activation function. Then, convolution operations, matrix averaging operations, and matrix dimension transformation operations are performed in sequence to obtain the pose transformation matrix.
[0149] By performing several convolution operations, key features in the spatial pose feature are extracted. After introducing nonlinear activation, the pose change matrix can be obtained through convolution operations, matrix averaging operations, and matrix dimension transformation operations. This pose change matrix is used to indicate the position of the camera in the target frame image, thereby estimating the motion trajectory.
[0150] In one implementation, the calculation formula for the above steps can be as follows:
[0151] F conv1 =ReLU(Conv1(F fout ))
[0152] F conv3 =ReLU(Conv3(ReLU(Conv3(F conv1 ))))
[0153] F pt =Conv3(F conv3 )
[0154] R,t=reshape(mean(0.01×F pt ))
[0155] In the formula, ReLU represents a nonlinear activation function, mean(·) represents the matrix averaging operation, reshape(·) represents the matrix dimension change operation R, and t represents the rotation and translation matrix, i.e., the pose change matrix.
[0156] As shown in the formula above, in this embodiment, a 1*1 convolution operation is performed first, followed by two 3*3 convolution operations, and activation is performed by the ReLU function after each convolution operation. In other embodiments, the number of convolution operations and the size of the convolution kernel can be adjusted based on the actual working conditions, and different nonlinear activation functions can be selected to achieve the effect of this embodiment.
[0157] Based on the above steps, the spatial attention mechanism efficiently integrates depth and semantic features, thereby enhancing the feature representation capability of pose estimation. While ensuring the algorithm's running speed, it significantly improves the pose estimation accuracy in practical application scenarios.
[0158] Meanwhile, since pose features are extracted based on the fusion of deep and semantic features, only one feature extraction backbone network structure needs to be set up, which greatly reduces the requirements for the feature extraction backbone network structure and improves the feature representation capability of the feature extraction backbone network, thereby ensuring the accuracy and running speed of the algorithm in practical application scenarios.
[0159] The training method of the deep learning model in this application is described in detail below. For specific details, please refer to [link / reference needed]. Figure 7 , Figure 7 This is a flowchart illustrating one implementation method of the deep learning model training method of this application.
[0160] like Figure 7 As shown, the training methods include:
[0161] S701. Construct a loss function based on the depth feature map and pose change matrix of the target frame image output by the deep learning model.
[0162] The loss function is composed of a weighted sum of the photometric loss between the reconstructed image and the target frame image, and the edge-aware depth smoothness loss of the depth feature map. The reconstructed image is constructed based on the depth feature map and pose change matrix of the target frame image.
[0163] In one implementation, the loss function can be calculated as follows:
[0164] L = L ph +λ1L sm
[0165] Among them, L ph For luminance loss, L sm The depth smoothness loss is used for edge awareness, and λ1 is the weight parameter set before training.
[0166] The calculation methods for each loss are explained in detail below:
[0167] The photometric loss between the reconstructed image and the target frame image consists of structural similarity (SSIM) and L1, and can be calculated as follows:
[0168]
[0169] In the formula, I t For the target frame image, I' t To reconstruct the image, I s For adjacent frame images, p is the index value of the pixel coordinates, and α is a hyperparameter that is obtained by preset, for example, it can be 0.85.
[0170] L ssim The formula for calculating it is as follows: SSIM is used to determine the structural similarity between the reconstructed image and the target frame image.
[0171] Edge-aware depth smoothness loss L sm It is used to generate smooth depth changes in continuous regions, while forcing sharp edge distributions at pixels with abrupt changes. The calculation formula can be as follows:
[0172]
[0173] In the formula, It is the mean-normalized inverse depth map of the depth feature map, where i,j represent the pixel index values of the target frame image. and These represent the gradients in the x and y directions of the target frame image, respectively.
[0174] S702. Based on the loss function, update the parameters of the deep neural network along the direction of gradient descent until the loss function converges.
[0175] By using the loss function described above for self-supervised training of the deep neural network, the accuracy of depth estimation and pose estimation of the algorithm is guaranteed.
[0176] This application also provides a lightweight self-supervised visual odometry architecture with a multi-feature fusion architecture; please refer to [link to relevant documentation]. Figure 8 , Figure 8 This is a schematic diagram of one embodiment of the lightweight self-supervised visual odometry with multi-feature fusion architecture of this application.
[0177] like Figure 8 As shown, the visual odometry includes an input module 21 and an output module 22;
[0178] The input module 21 is used to acquire the target frame image and its neighboring frame images and input them into the pre-trained deep learning model.
[0179] The output module 22 is used to obtain the depth feature map, semantic feature map and pose transformation matrix of the target frame image based on the output of the deep learning model.
[0180] The deep learning model includes a deep semantic prediction network and a pose prediction network. The deep semantic prediction network is used to concatenate the target frame image and adjacent frame images and extract features to obtain a first feature map. Deep features and semantic features are extracted from the first feature map, and deep feature maps and semantic feature maps are output respectively.
[0181] The pose prediction network is used to fuse the depth feature map and the semantic feature map and extract pose features to obtain a second feature map; features are extracted from the second feature map and the pose change matrix is output.
[0182] In one embodiment, a training module 23 is further included. The training module 23 is used to construct a loss function based on the depth feature map and pose change matrix of the target frame image output by the deep learning model; and based on the loss function, update the parameters of the deep neural network along the direction of gradient descent until the loss function converges.
[0183] The loss function is composed of a weighted sum of the photometric loss between the reconstructed image and the target frame image, and the edge-aware depth smoothness loss of the depth feature map. The reconstructed image is constructed based on the depth feature map and pose change matrix of the target frame image.
[0184] As referred above Figures 1 to 7 The visual odometry method according to embodiments of this specification has been described. The details mentioned in the above description of the method embodiments also apply to the visual odometry of the embodiments of this specification. The above-described visual odometry can be implemented in hardware, software, or a combination of hardware and software.
[0185] This application also provides an electronic device, please refer to... Figure 9 , Figure 9 This is a schematic diagram of one embodiment of the electronic device of this application. For example... Figure 9 As shown, the electronic device 30 may include at least one processor 31, a memory 32 (e.g., non-volatile memory), a RAM 33, and a communication interface 34, and the at least one processor 31, memory 32, RAM 33, and communication interface 34 are connected together via a bus 35. The at least one processor 31 executes at least one computer-readable instruction stored or encoded in the memory 32.
[0186] It should be understood that the computer-executable instructions stored in memory 32, when executed, cause at least one processor 31 to perform the above-described combinations in the various embodiments of this specification. Figures 1-5 The description includes various operations and functions.
[0187] In the embodiments of this specification, electronic device 30 may include, but is not limited to: personal computer, server computer, workstation, desktop computer, laptop computer, notebook computer, mobile electronic device, smartphone, tablet computer, cellular phone, personal digital assistant (PDA), handheld device, messaging device, wearable electronic device, consumer electronic device, etc.
[0188] According to one embodiment, a program product, such as a machine-readable medium, is provided. The machine-readable medium may have instructions (i.e., the elements implemented in software as described above), which, when executed by a machine, cause the machine to perform the above-described combinations of the various embodiments of this specification. Figures 1-5 The various operations and functions described. Specifically, a system or apparatus equipped with a readable storage medium storing software program code that implements the functions of any of the embodiments described above, and enabling the computer or processor of the system or apparatus to read and execute the instructions stored in the readable storage medium.
[0189] In this case, the program code read from the readable medium itself can perform the functions of any of the above embodiments, and therefore the machine-readable code and the readable storage medium storing the machine-readable code constitute a part of this specification.
[0190] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer or the cloud via a communication network.
[0191] Those skilled in the art will understand that the various embodiments disclosed above can be modified and varied without departing from the spirit of the invention. Therefore, the scope of protection of this specification should be defined by the appended claims.
[0192] It should be noted that not all steps and units in the above process and system structure diagrams are mandatory; some steps or units can be omitted according to actual needs. The execution order of each step is not fixed and can be determined as needed. The device structure described in the above embodiments can be a physical structure or a logical structure. That is, some units may be implemented by the same physical client, or some units may be implemented by multiple physical clients, or they may be jointly implemented by certain components in multiple independent devices.
[0193] In the above embodiments, the hardware units or modules can be implemented mechanically or electrically. For example, a hardware unit, module, or processor may include permanent dedicated circuitry or logic (such as a dedicated processor, FPGA, or ASIC) to perform the corresponding operation. The hardware unit or processor may also include programmable logic or circuitry (such as a general-purpose processor or other programmable processor), which can be temporarily configured by software to perform the corresponding operation. The specific implementation method (mechanical, dedicated permanent circuitry, or temporarily configured circuitry) can be determined based on cost and time considerations.
[0194] The specific embodiments described above with reference to the accompanying drawings are exemplary embodiments, but do not represent all embodiments that can be implemented or fall within the scope of the claims. The term "exemplary" as used throughout this specification means "serving as an example, instance, or illustration" and does not imply that it is "preferred" or "advantageous" compared to other embodiments. Specific details are included to provide an understanding of the described techniques. However, these techniques can be practiced without these specific details. In some instances, well-known structures and apparatuses are shown in block diagram form to avoid obscuring the concepts of the described embodiments.
[0195] The foregoing description of this disclosure is provided to enable any person skilled in the art to implement or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles applicable herein can be applied to other variations without departing from the scope of this disclosure. Therefore, this disclosure is not limited to the examples and designs described herein, but is consistent with the widest scope of the principles and novel features disclosed herein.
Claims
1. A lightweight self-supervised visual odometry algorithm with a multi-feature fusion architecture, characterized in that, include: Acquire the target frame image and its neighboring frame images, and input them into a pre-trained deep learning model; Based on the output of the deep learning model, the depth feature map, semantic feature map, and pose change matrix of the target frame image are obtained. The deep learning model includes a deep semantic prediction network and a pose prediction network. The deep semantic prediction network is used to concatenate the target frame image and the adjacent frame images and extract features to obtain a first feature map. Depth features and semantic features are extracted from the first feature map, and the depth feature map and the semantic feature map are output respectively. The pose prediction network is used to fuse the depth feature map and the semantic feature map and extract pose features to obtain spatial pose features. Features are extracted from the spatial pose features, and the pose change matrix is output.
2. The lightweight self-supervised visual odometry method according to claim 1, characterized in that, The steps for extracting depth features from the first feature map include: The first feature map is subjected to convolution and upsampling operations in sequence to obtain a first depth key feature map with the same resolution as the first feature map. The first feature map and the first depth key feature map are concatenated, and then a convolution operation is performed to obtain the first depth fusion feature map. The first deep fusion feature map is upsampled and then activated by a nonlinear activation function to obtain the deep feature map.
3. The lightweight self-supervised visual odometry method according to claim 1, characterized in that, The steps for extracting semantic features from the first feature map include: The first feature map is convolved, activated by a non-linear activation function, and then upsampled to obtain a first semantic key feature map with the same resolution as the first feature map. The first semantic key feature map and the first feature map are added together to obtain the first semantic fusion feature map; The first semantic fusion feature map is convolved and then activated by a non-linear activation function to obtain the semantic feature map.
4. The lightweight self-supervised visual odometry method according to claim 1, characterized in that, The step of fusing the depth feature map and the semantic feature map and extracting pose features includes: The depth feature map and the semantic feature map are concatenated, and then max pooling and average pooling operations are performed respectively. The results of the two operations are concatenated again to obtain the second fused feature map. The second fused feature map is convolved and then activated by a non-linear activation function to obtain the second feature map. The second feature map is subjected to convolution and batch normalization operations in sequence, and then activated by a non-linear activation function to obtain a spatially enhanced feature map. The second feature map and the spatial enhancement feature map are fused to obtain spatial pose features.
5. The lightweight self-supervised visual odometry method according to claim 1, characterized in that, The step of extracting features from the spatial pose features and outputting the pose change matrix includes: The spatial pose features are subjected to several convolution operations, and after each convolution operation, they are activated by a nonlinear activation function. Then, convolution operations, matrix averaging operations, and matrix dimension transformation operations are performed in sequence to obtain the pose transformation matrix.
6. The lightweight self-supervised visual odometry method according to claim 5, characterized in that, The steps of performing several convolution operations on the spatial pose features, activating them with a non-linear activation function after each convolution operation, and then sequentially performing convolution operations, matrix averaging operations, and matrix dimension transformation operations to obtain the pose transformation matrix are as follows: F c1 =ReLU(Conv1(F fout )) F c3 =ReLU(Conv3(ReLU(Conv3(F c1 )))) F p =Conv3(F c3 ) R,t=reshape(mean(0.01×F p )) In the formula, F fout For the spatial pose features, ReLU is a non-linear activation function, mean(·) is a matrix averaging operation, reshape(·) is a matrix dimension change operation, R,t is a rotation and translation matrix, i.e., a pose change matrix, Conv1 is a 1*1 convolution operation, and Conv3 is a 3*3 convolution operation.
7. The lightweight self-supervised visual odometry method according to claim 1, characterized in that, The training methods for the deep learning model include: Based on the depth feature map and pose change matrix of the target frame image output by the deep learning model, a loss function is constructed. Based on the loss function, the parameters of the deep neural network are updated along the direction of gradient descent until the loss function converges; The loss function is composed of a weighted sum of the photometric loss between the reconstructed image and the target frame image, and the edge-aware depth smoothness loss of the depth feature map. The reconstructed image is constructed based on the depth feature map of the target frame image and the pose change matrix.
8. The lightweight self-supervised visual odometry method according to claim 7, characterized in that, The formula for calculating the photometric loss is as follows: In the formula, I t For the target frame image, I' t For the reconstructed image, I s For the adjacent frame images, p is the index value of the pixel coordinates, α is a hyperparameter, and L... ssim For structural similarity; and / or, The formula for calculating the edge-aware depth smoothness loss is as follows: In the formula, It is the mean-normalized inverse depth map of the depth feature map, where i,j represent the pixel index values of the target frame image. and These represent the gradients in the x and y directions of the target frame image, respectively.
9. A lightweight self-supervised visual odometry architecture with a multi-feature fusion architecture, characterized in that, include: The input module is used to acquire the target frame image and its neighboring frame images and input them into the pre-trained deep learning model. The output module is used to obtain the depth feature map, semantic feature map, and pose transformation matrix of the target frame image based on the output of the deep learning model. The deep learning model includes a deep semantic prediction network and a pose prediction network. The deep semantic prediction network is used to concatenate the target frame image and the adjacent frame images and extract features to obtain a first feature map. Depth features and semantic features are extracted from the first feature map, and the depth feature map and the semantic feature map are output respectively. The pose prediction network is used to fuse the depth feature map and the semantic feature map and extract pose features to obtain a second feature map; features are extracted from the second feature map and the pose change matrix is output.
10. An electronic device, comprising: At least one processor; as well as A memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform a lightweight self-supervised visual odometry algorithm based on a multi-feature fusion architecture as described in any one of claims 1 to 8.
11. A machine-readable storage medium storing executable instructions that, when executed, cause the machine to perform a lightweight self-supervised visual odometry based on a multi-feature fusion architecture as described in any one of claims 1 to 8.