Unmanned system-oriented adaptive deep learning visual odometry positioning method and system

Abstract

The present application relates to the field of robot perception, and more particularly to an adaptive deep learning visual odometry positioning method and system for unmanned systems, which comprises receiving an RGB image to be measured, obtaining a previous frame image from a history record for the current image frame to be measured, and composing two adjacent RGB images as positioning input data; inputting the positioning input data into a trained positioning model, extracting spatio-temporal features through an adaptive convolution module, and outputting the relative pose of the camera by a pose estimation network. The method solves the problem of insufficient generalization ability and poor positioning accuracy of current visual odometry positioning methods in complex scenes, and makes the robot and autonomous driving equipment more accurate and stable in positioning in complex scenes, and more adaptable than existing methods.

Description

Technical Field

[0001] This application relates to the fields of robot perception and precise positioning for autonomous driving, and in particular to an adaptive deep learning visual odometry method and system for unmanned systems. Background Technology

[0002] Visual odometry (VO) is a core technology for autonomous driving and robot navigation, used to provide real-time positioning information. VO methods are broadly classified into two categories: geometry-based and learning-based. Geometry-based methods, such as ORB-SLAM3 and VINS, estimate camera motion by establishing geometric constraints using visual features matched across consecutive frames. These methods are known for their solid theoretical foundation, interpretability, and high accuracy in structured environments, but they also have limitations, such as reliance on high-quality feature extraction, sensitivity to matching errors, and insufficient generalization ability in complex or dynamic scenes.

[0003] In contrast, learning-based VOs, such as TartanVO and DeepVO, utilize deep neural networks (DNNs) to directly regress camera motion from raw visual input. These methods do not rely on hand-designed geometric models but instead autonomously learn motion states and scene representations from the data. Compared to geometry-based methods, learning-based VOs exhibit stronger robustness under challenging conditions (such as low-texture or dynamic scenes) and demonstrate better generalization ability to new environments after training on diverse datasets. However, such methods typically require large amounts of training data and are prone to performance degradation in highly dynamic scenes or unknown environments deviating from the training distribution. Summary of the Invention

[0004] To address the shortcomings of current visual odometry methods in generalizing to complex scenarios and their poor positioning accuracy, this application provides an adaptive deep learning visual odometry method and system for unmanned systems. By simulating real-world interference through data augmentation and time-adjusted adaptive algorithms, the model's generalization ability is improved, enabling robots and autonomous driving devices to achieve more accurate and stable positioning in complex scenarios (such as rapid movement, changing lighting, and occlusion), demonstrating greater adaptability than existing methods.

[0005] Firstly, this application provides an adaptive deep learning visual odometry localization method for unmanned systems:

[0006] Receive the RGB image to be tested, and for the current frame of the image to be tested, obtain the previous frame image from the history record to form two adjacent RGB images as the positioning input data.

[0007] The positioning input data is input into the trained positioning model. After the spatiotemporal features are extracted by the adaptive convolution module, the relative pose of the camera is output by the pose estimation network.

[0008] The localization model is trained in the following manner:

[0009] RGB images containing real pose labels are received sequentially, and two adjacent RGB images are selected from the received images as training input data.

[0010] The training input data is subjected to data augmentation processing, which includes motion blur enhancement and basic data augmentation. The motion blur enhancement is used to simulate camera motion artifacts, and the basic data augmentation is used to simulate changes in environmental conditions and occlusion.

[0011] A localization model containing an adaptive convolutional module and a pose estimation network is constructed. The adaptive convolutional module generates calibration weights by combining the spatial and temporal contexts of adjacent frames, and dynamically adjusts the convolutional kernel weights to extract spatiotemporal features from the data-enhanced image.

[0012] Based on the aforementioned spatiotemporal features, a pose estimation network outputs a predicted pose. The mean square error between the true pose and the predicted pose is used as the loss function to iteratively optimize the localization model parameters until the loss function converges.

[0013] By adopting the above technical solutions, data augmentation is used to expand sample diversity during training, enabling the model to adapt to complex scenarios such as motion blur and lighting changes. During localization, adaptive feature extraction is used to achieve continuous and real-time estimation of camera motion, solving the problems of insufficient generalization ability and weak inter-frame correlation modeling in existing technologies, and improving the overall localization accuracy and robustness.

[0014] Optionally, the motion blur includes radial blur and / or directional blur;

[0015] The radial blur is caused by the translational motion of the camera along the roll axis. Specifically, it is achieved by determining the center point of the image, causing the pixels to shift according to their distance from the center point, and mixing inward and outward blurred sub-images.

[0016] The directional blur is caused by a combination of translation and rotation of the camera along the yaw and pitch axes. Specifically, it is achieved by simulating with a Gaussian kernel and mixing the pixel with its neighboring pixels in the opposite direction of the motion to produce a trailing effect.

[0017] Optionally, the radial blurring includes generating a sub-image by calculating the pixel value of the target position in the displacement map through linear interpolation, combining the inward and outward blurred sub-images using weights, and repeating the process a preset number of iterations to produce a radial blurring effect.

[0018] Optionally, the directional blurring is achieved by applying an affine transformation to the Gaussian kernel using a rotation matrix, aligning the blur direction with the motion direction, wherein the rotation matrix is ​​calculated based on the Euler angles corresponding to the camera motion.

[0019] Optionally, the basic data augmentation includes color jitter and random erasure; the color jitter includes simulating changes in lighting or environmental conditions in the real world by randomly perturbing the color attributes of the image during training; the random erasure simulates occlusion by randomly covering rectangular areas in the image, which is used to simulate the situation where objects are partially occluded in real-world scenes.

[0020] Optionally, the color attribute includes at least one of brightness, contrast, and saturation.

[0021] Optionally, the kernel weight adjustment method of the adaptive convolution module includes:

[0022] The convolution kernel weight W at time t t The benchmark weight W β With calibration weight α t The product of; wherein the calibration weight α t The description vector is obtained by performing global average pooling on the input image, and then generated by stacked one-dimensional convolution.

[0023] Optionally, the pose estimation network is based on the Transformer architecture, which divides the feature map after adaptive convolution into image blocks, combines them with position encoding, processes them through a Transformer encoder, and then outputs a 6-DOF relative pose through a multilayer perceptron.

[0024] Optionally, the loss function includes:

[0025]

[0026] Where B is the batch size (set to 4 in this study), y b,i Represents the true relative pose. This indicates the predicted relative pose.

[0027] Secondly, this application provides an adaptive deep learning visual odometry localization system for unmanned systems; the system includes modules for performing the methods in the first aspect or any possible implementation of the first aspect:

[0028] The system includes:

[0029] The receiving module is used to receive the RGB image to be tested. For the current image frame to be tested, it retrieves the previous frame image from the historical records and forms two adjacent RGB images as positioning input data.

[0030] The localization model is used to receive the localization input data, extract spatiotemporal features through an adaptive convolution module, and output the relative pose of the camera by a pose estimation network.

[0031] The localization model is trained in the following manner:

[0032] The system sequentially receives RGB images containing real pose labels, and selects two adjacent RGB images from the received images as training input data. Data augmentation processing is performed on the training input data, including motion blur enhancement and basic data augmentation. Motion blur enhancement is used to simulate camera motion artifacts, and basic data augmentation is used to simulate changes in environmental conditions and occlusion. A localization model is constructed, comprising an adaptive convolutional module and a pose estimation network. The adaptive convolutional module generates calibration weights by combining the spatial and temporal contexts of adjacent frames, dynamically adjusting the convolutional kernel weights to extract spatiotemporal features from the data-augmented images. Based on the spatiotemporal features, the pose estimation network outputs a predicted pose. The mean square error between the real pose and the predicted pose is used as the loss function, and the localization model parameters are iteratively optimized until the loss function converges.

[0033] Thirdly, this application provides a computer device including a processor, a memory, and a communication bus, wherein the communication bus is used to realize a communication connection between the processor and the memory, and the processor is used to execute a computer program stored in the memory to implement the method described in any of the preceding claims.

[0034] Fourthly, this application also provides a computer-readable storage medium storing a computer program; the computer program can be executed by a processor to implement the method described above.

[0035] Fifthly, this application also provides a computer program product, including a computer program that can be executed by a processor to implement the method described above.

[0036] This application includes at least the following beneficial technical effects:

[0037] 1. By simulating real-world interference through data augmentation and time-adjusted adaptive algorithms, the generalization ability of the model is improved, enabling robots and autonomous driving devices to locate more accurately and stably in complex scenarios, making them more adaptable than existing methods. Attached Figure Description

[0038] Figure 1 This is a schematic flowchart of a visual odometry method provided in an embodiment of this application;

[0039] Figure 2 This is a schematic diagram of a localization model training process provided in an embodiment of this application;

[0040] Figure 3 A visual odometry localization algorithm architecture diagram provided in this application embodiment;

[0041] Figure 4 A comparison image of an image before and after radial blur enhancement is provided in an embodiment of this application;

[0042] Figure 5 This application provides an image comparison diagram before and after applying radial blur enhancement (different kernel sizes);

[0043] Figure 6 A trajectory map of a representative sequence estimated using different methods is provided as an embodiment of this application;

[0044] Figure 7 An average error result and variance plot of a test sequence provided in an embodiment of this application;

[0045] Figure 8 An experimental result diagram of a different method on a KITTI dataset test sequence provided in the embodiments of this application;

[0046] Figure 9 Experimental results of pose estimation time per frame for different methods provided in this application embodiment;

[0047] Figure 10 A comparison diagram of results for different VOs on sequences 09 and 10 provided in this application embodiment;

[0048] Figure 11 This application provides an embodiment of the algorithm using different enhancements and with or without adaptive convolution, resulting in a graph.

[0049] Figure 12 An effect diagram of using motion blur to enhance probability is provided in an embodiment of this application;

[0050] Figure 13 This application provides an example of the effect of using different radial ambiguity coefficients.

[0051] Figure 14 This is a schematic diagram of the visual odometry system structure provided in the embodiments of this application;

[0052] Figure 15 This is a schematic diagram of the computer device structure provided in an embodiment of this application. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0054] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and includes any or all possible combinations of one or more of the listed items. The term “exemplary” means “serving as an example, embodiment, or illustration,” and any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments. The terms “first” and “second” are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as “first” or “second” may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, “a plurality” means two or more.

[0055] refer to Figure 1 An adaptive deep learning visual odometry localization method for unmanned systems mainly includes the following steps:

[0056] S102: Receive the RGB image to be tested, and for the current frame of the image to be tested, obtain the previous frame image from the historical record and form two adjacent RGB images as positioning input data.

[0057] S104: Input the localization input data into the trained localization model. After extracting the spatiotemporal features through the adaptive convolution module, the pose estimation network outputs the relative pose of the camera.

[0058] refer to Figure 2 The localization model is trained in the following way, mainly including the following steps:

[0059] S202: Receive RGB images containing real pose labels in sequence, and select two adjacent RGB images from the received images as training input data;

[0060] S204: Perform data augmentation on the training input data; data augmentation includes motion blur enhancement and basic data augmentation, where motion blur enhancement is used to simulate camera motion artifacts, and basic data augmentation is used to simulate changes in environmental conditions and occlusion;

[0061] While conventional methods can alleviate the problems of data scarcity and insufficient robustness in network models through data augmentation, conventional image augmentation techniques (such as flipping and rotating) often disrupt the spatiotemporal consistency of continuous visual data, rendering the model unable to use this training data. To address this, this embodiment increases data diversity by simulating motion blur, seasonal changes, and occlusion in the training sequence.

[0062] Optionally, motion blur includes radial blur and / or directional blur. Radial blur is caused by the camera's translational motion along the roll axis, specifically achieved by determining the image center point, causing pixels to shift according to their distance from the center point, and blending inward and outward blurred sub-images. Directional blur is caused by a combination of translational and rotational motion of the camera along the yaw and pitch axes, specifically achieved by simulating with a Gaussian kernel and blending pixels with their neighboring pixels in the opposite direction of motion to produce a trailing effect.

[0063] Optionally, radial blurring involves generating sub-images by calculating pixel values ​​at a specified target location using linear interpolation, combining inward and outward blurred sub-images using weights, and repeating this process a preset number of iterations to produce a radial blurring effect. Directional blurring applies an affine transformation to a Gaussian kernel using a rotation matrix to align the blur direction with the motion direction, where the rotation matrix is ​​calculated based on the Euler angles corresponding to the camera motion.

[0064] Optionally, the basic data augmentation includes color jitter and random erasure; color jitter involves randomly perturbing the color attributes of the image during training to simulate changes in lighting or environmental conditions in the real world; random erasure simulates occlusion by randomly covering rectangular areas in the image to simulate the situation where objects are partially occluded in real-world scenes.

[0065] Optionally, the color attributes include at least one of brightness, contrast, and saturation.

[0066] S206: Construct a localization model that includes an adaptive convolutional module and a pose estimation network. The adaptive convolutional module combines the spatial and temporal contexts of adjacent frames to generate calibration weights and dynamically adjusts the convolutional kernel weights to extract spatiotemporal features from the data-augmented image.

[0067] Existing odometry methods rely on the assumption of spatial translation invariance, sharing convolutional weights across different frames, thus losing the ability to model the temporal dimension. To address this, this embodiment considers the temporal correlation between frames, dynamically calibrating the weights of the convolutional kernels instead of keeping them constant, thereby improving the network model's ability to correlate contextually and the accuracy of pose estimation.

[0068] Optionally, the kernel weight adjustment methods of the adaptive convolution module include:

[0069] The convolution kernel weight W at time tt The benchmark weight W β With calibration weight α t The product of; where the calibration weight α t The description vector is obtained by performing global average pooling on the input image, and then generated by stacked one-dimensional convolution.

[0070] Optionally, the pose estimation network is based on the Transformer architecture, which divides the feature map after adaptive convolution into image blocks, combines them with position encoding, processes them through a Transformer encoder, and then outputs a 6-DOF relative pose through a multilayer perceptron.

[0071] S208: Based on spatiotemporal features, the pose estimation network outputs the predicted pose, and the mean square error between the true pose and the predicted pose is used as the loss function to iteratively optimize the localization model parameters until the loss function converges.

[0072] Optionally, the loss function includes:

[0073]

[0074] Where B is the batch size (set to 4 in this study), y b,i Represents the true relative pose. This indicates the predicted relative pose.

[0075] Optionally, this visual odometry method can be based on, for example, Figure 3 The illustrated algorithm (Data Augmentation and Adaptive Transformer-based VO, RADVO) architecture comprises two main modules: a data augmentation module and an adaptive Transformer-based VO module. The data augmentation module enhances data diversity by simulating rapid camera movement, weather changes, and occlusion. The adaptive Transformer-based VO module estimates the camera pose by combining adaptive convolutions with popular Transformer networks.

[0076] Motion blur enhancement aims to simulate highly dynamic scenes, making models robust to motion blur that occurs in the real world. For example, cameras mounted on robots or vehicles often capture blurred images when moving rapidly. If a model is trained only on sharp images, it may fail when faced with motion blur. In this embodiment, two motion blur enhancement strategies are proposed: radial blur and directional blur.

[0077] Radial blur is typically caused by the translational motion of the camera along the roll axis. This translational motion can be derived from real data. In continuous camera motion, the relative pose transformation T∈SE(3) between two frames. NRepresenting motion, including the rotation matrix R∈SE(3) N Translation vector t∈R 3×1 By applying different scaling factors to adjust the camera's movement speed, data with varying degrees of blur can be obtained. It's worth noting that the camera's sampling frequency remains constant throughout this process.

[0078] The radial blur operator is a technique that offsets image pixels based on the distance between a pixel and a specified center point, producing a blurring effect that radiates outwards. To achieve this effect, this embodiment introduces two key parameters: a blur intensity parameter (blur), used to control the intensity of the blur effect, such as... Figure 4 As shown; the iteration parameter `iterations` determines the number of times the blur operation is performed. Let the width and height of the image be `w` and `h` respectively, then the coordinates of the image center point are `center`. x =h / 2 and center y =w / 2, the calculation formula is as follows:

[0079] blur=t roll *η, iterations=C,C∈[1,M], (1)

[0080] Among them, t roll The translation along the roll axis is obtained from the relative transformation matrix T; η is a noise factor randomly generated in the interval [0,1]; C is a random integer ranging from 1 to M, where M is a constant.

[0081] For a pixel located at (i,j) (where i is the row index and j is the column index), the displacement caused by the blur effect can be decomposed into horizontal and vertical components. The position after displacement is represented as Map[i,j]:

[0082]

[0083] Where, Δ i and Δ j These represent the offsets in the row and column directions, respectively. The ± sign determines the fuzzy direction: a positive sign indicates outward diffusion, and a negative sign indicates inward contraction. Combined with the fuzzy coefficients, the displacement can be further expressed as:

[0084] Map[i,j]=[i+Δ i *blur,j+Δ j *blur]. (3)

[0085] By calculating Map[i,j] for all pixel positions, a displacement matrix Map can be constructed. xy ∈R h×w This matrix encapsulates the image's displacement in the height and width dimensions, expressed as:

[0086]

[0087] According to the offset term Δ i and Δ j The choice of ± symbol can generate two different displacement diagrams: Indicates outward blurring. It indicates inward ambiguity.

[0088] Subsequently, interpolation methods are used to calculate the pixel values ​​at the specified target locations in these displacement maps, generating two sub-images. These two sub-images are then combined using weights w1 and w2.

[0089]

[0090] The map(·) function uses linear interpolation with weights set to w1 = w2 = 0.5. This process is repeated iterations times to gradually produce a radial blur effect, expressed as:

[0091]

[0092] Orientation blur is typically caused by a combination of translational and rotational motions along the yaw and pitch axes. This embodiment uses a Gaussian kernel to simulate orientation blur, producing a trailing effect by blending a pixel with its neighboring pixels in the opposite direction of motion, such as... Figure 5 As shown. Its implementation process is as follows:

[0093] First, create a Gaussian kernel with an initial size of odd number, and the default blur direction is horizontal. Its expression is:

[0094]

[0095] Where i and j are the row and column indices, respectively. Then, to align the blur direction with the motion direction, an affine transformation is applied to the Gaussian kernel using the rotation matrix R. This rotation matrix can be decomposed into Euler angles φ, θ, ψ. The two-dimensional rotation matrix used for the Gaussian kernel is then calculated using the following formula:

[0096]

[0097] To approximate the main direction of camera movement, the dominant angle can be selected:

[0098] angle=max(|φ|,|θ|,|ψ|), (9)

[0099] The absolute value ensures the selection of the largest rotation amplitude, regardless of its sign. Based on this dominant angle, the affine transformation matrix Rot(angle) is calculated:

[0100]

[0101] Finally, the Gaussian kernel is transformed according to this affine transformation matrix:

[0102] kernel′=Wrap(kernel,Rot(angle)), (11)

[0103] Here, Wrap represents an affine transformation operation (e.g., warping or resampling the kernel mesh). The transformed kernel′ reflects the fuzzy direction.

[0104] This embodiment also considers basic enhancements including color dithering and random erasure. Color dithering enhancement simulates changes in lighting or environmental conditions in the real world by randomly perturbing the color attributes of an image (such as brightness, contrast, and saturation) during training. This forces the model to learn robust features invariant to color distortion, thereby improving the generalization ability of VO.

[0105] Random erasure is used to simulate situations where objects are partially occluded in real-world scenes (such as obstacles, lighting, or sensor noise). Controlled occlusion is introduced by randomly masking rectangular regions of different sizes, shapes, and locations in the training images. This encourages the VO model to learn robust and diverse feature representations, avoiding over-reliance on specific visual patterns or regions in the data.

[0106] The adaptive Transformer-based pose estimation (VO) model comprises two modules: an adaptive convolutional module and a pose estimation network. Previous learning-based VO work typically uses shared network weights across all spatial locations and temporal frames. This can affect the effectiveness of extracting spatiotemporal features, thus reducing VO accuracy. Therefore, this embodiment introduces adaptive convolution to better capture inter-frame correspondences. The weights of the adaptive convolutional kernel are no longer fixed across different frames, but are adjusted by combining the original weights with a calibration weight α. t The weights are dynamically adjusted through multiplication. Mathematically, the weight w at time t is... t Decomposed into benchmark weights W b With calibration weight α t The product of:

[0107] W t =α t ×W b (12)

[0108] Wherein, the calibration weight α t Based on the input data and the baseline weight W b Adaptively generated across all channels. To capture spatiotemporal information, weight α... t The spatial and temporal contexts of adjacent frames must be considered simultaneously. The adaptive convolutional module takes the enhanced image as input and outputs a feature map. For each frame, by processing the input image x... tGlobal average pooling (GAP) is performed on the spatial dimension to obtain the description vector v. t =GAP(x t )∈R T×C Subsequently, calibration weights are generated by applying two stacked one-dimensional convolutional layers. This process can be represented as:

[0109]

[0110] Where Conv represents a convolutional layer, Act is the ReLU activation function, and BN represents batch normalization. To gain a broader perspective, this embodiment incorporates the global descriptor g into the weight generation process through a linear mapping function FC, further integrating global temporal information. Weight α t It can be represented as:

[0111]

[0112] Here, g is obtained through global average pooling in both time and space dimensions.

[0113] The pose estimation network, developed based on TimeSFormer, takes the feature map output by the adaptive convolutional module as input and outputs a 6-DOF pose. Specifically, the input feature map is first divided into... Each block is a non-overlapping p×p block. Then, at time step t, each block... (where p∈{1,...,N} is vectorized and linked by the embedding matrix E and the learnable positional encoding E) pos Combined with the classification marker cls, a sequence is formed. The sequence is then processed by a Transformer encoder, which employs a computationally efficient spatiotemporal self-attention mechanism. This mechanism models spatial intra-frame correlations and temporal inter-frame dependencies by decomposing the attention head. The resulting spatiotemporal feature representation is progressively refined through multilayer perceptron (MLP) projection, ultimately generating an accurate 6-DoF pose estimate. For training the network, this embodiment uses the mean squared error (MSE) between the true pose and the predicted pose as the loss function.

[0114]

[0115] Where B is the batch size (set to 4 in this embodiment), y b,i This represents the true 6-DoF relative pose. This indicates the predicted relative pose.

[0116] In this embodiment, the learning rate is set to 0.00001, the training epochs are 100, and the optimizer is Adam with its default parameters. For the data augmentation hyperparameters, the augmentation probability is 0.1. For radial fuzziness, the maximum number of iterations is 8.

[0117] Based on the technical solution provided in the embodiments of this application, a data augmentation regularization method is added compared with the existing method. This is a data augmentation method suitable for visual odometry. The motion blur enhancement technology designed specifically for visual odometry simulates challenging scenarios in the real world (such as sudden camera movement), forcing the network to learn feature representations that are invariant to blur, thereby improving robustness to motion degradation.

[0118] Furthermore, by incorporating an adaptive Transformer architecture, the proposed method explicitly learns the geometric and temporal relationships in visual data, and adaptively adjusts the convolution weights to better capture spatiotemporal features, thus addressing environmental dynamics. Therefore, it achieves superior performance compared to other methods. The effectiveness of the proposed method is evaluated using the publicly available KITTI dataset, and the evaluation process includes the following comparisons:

[0119] The results of the proposed method and benchmark methods in this invention were compared on the KITTI (an internationally recognized authoritative autonomous driving dataset). The benchmark methods compared included ORB-SLAM3, DeepVO, TSformer-VO, and SWformer-VO. Sequences 00, 02, 08, and 09 were used for training, and the remaining sequences were used for testing. Figure 6 It can be seen that the method of this embodiment of the invention is significantly more competitive than all benchmark methods on multiple sequences. Figure 7 Except for the average relative rotation error, which ranks second, the test errors of the methods in this embodiment are lower than all other benchmark methods.

[0120] Detailed quantitative results were also compared with the baseline method on each test sequence, such as... Figure 8 As shown, RADVO ranks first in almost all metrics, only slightly inferior to ORB-SLAM3 in average rotational error.

[0121] The computational costs of this invention and benchmark methods were also compared. Figure 9 As shown, this invention ranks second in pose estimation time per frame, second only to TSformer-VO. RADVO uses the same backbone network as TSformer-VO, but RADVO introduces adaptive convolutional layers, thus taking more time. However, RADVO remains competitive compared to other methods.

[0122] We compared our approach with popular unsupervised learning VO methods, including SFMlearner, Depth-VO-Feat, GeoNet, Bian et al., Masked GANs, and Wang et al. Figure 10 The results show that the method proposed in this embodiment of the invention performs best in terms of average translation error, while it is at a moderate level in terms of average rotation error. Although it does not consider geometric and scale consistency constraints, it is still competitive.

[0123] This embodiment also conducts an ablation study on the proposed method, first analyzing the effects of using different enhancement strategies and whether or not an adaptive convolution module is included. Figure 11 This indicates that each component improves performance to some extent, and the maximum gain is obtained when all components are combined, where MB represents motion blur, AC is adaptive convolution, and BA is the basic enhancement strategy.

[0124] Set the probability of using motion blur enhancement to a value between 0.1 and 0.5. Figure 12 The results show that most performance metrics decline as the probability of use increases. This can be explained by the fact that applying data augmentation too frequently can severely disrupt the data distribution. Therefore, higher probabilities are not always better, and appropriate values ​​should be chosen based on the characteristics of the dataset. The degree of motion blur augmentation is determined by the blur coefficients and the number of iterations. In practice, the number of iterations is often randomized to enhance data diversity. In the experiments, the number of iterations was fixed at 8, and the blur coefficients were tested in the range of [0.001, 0.005]. Figure 13 The results show that as the coefficient increases to 0.002, most performance metrics initially decrease, then generally increase, indicating a degradation in accuracy. From a visual perception perspective, when the coefficient reaches 0.004, the image becomes significantly blurred, making pose estimation difficult. Therefore, choosing appropriate coefficient values ​​is crucial for achieving satisfactory performance.

[0125] Based on the above method embodiments, this embodiment also provides an adaptive deep learning visual odometry localization system for unmanned systems, which can be used to implement the steps of the above method.

[0126] refer to Figure 14 The visual odometry system includes:

[0127] The receiving module 141 is used to receive the RGB image to be tested, and for the current image frame to be tested, to obtain the previous frame image from the historical record and form two adjacent RGB images as positioning input data.

[0128] The localization model 142 is used to receive localization input data, extract spatiotemporal features through an adaptive convolution module, and output the relative pose of the camera by the pose estimation network.

[0129] The localization model is trained as follows: RGB images containing real pose labels are received sequentially, and two adjacent RGB images are selected as training input data. Data augmentation processing is performed on the training input data, including motion blur enhancement and basic data augmentation. Motion blur enhancement is used to simulate camera motion artifacts, and basic data augmentation is used to simulate changes in environmental conditions and occlusion. A localization model containing an adaptive convolutional module and a pose estimation network is constructed. The adaptive convolutional module combines the spatial and temporal contexts of adjacent frames to generate calibration weights, and dynamically adjusts the convolutional kernel weights to extract spatiotemporal features from the data-augmented images. Based on the spatiotemporal features, the pose estimation network outputs the predicted pose. The mean square error between the real pose and the predicted pose is used as the loss function, and the localization model parameters are iteratively optimized until the loss function converges.

[0130] Various variations and specific examples of the methods provided in the above embodiments are also applicable to the system of this embodiment. Through the foregoing detailed description of the methods, those skilled in the art can clearly understand the implementation method of the system in this embodiment. For the sake of brevity, they will not be described in detail here.

[0131] To better execute the above methods, embodiments of this application also provide a computer device, such as... Figure 15 As shown, the computer device includes a processor 151, a memory 152, and a communication bus 153 for enabling communication between the processor 151 and the memory 152.

[0132] Computer devices can take many forms, including mobile phones, tablets, handheld computers, laptops, and desktop computers.

[0133] The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for at least one function, and instructions for implementing the methods provided in the above embodiments; the data storage area may store data involved in the methods provided in the above embodiments.

[0134] Optionally, the memory may be a read-only memory (ROM), random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), optical disc (including compact disc read-only memory (CD-ROM), compressed optical disc, laser disc, digital versatile optical disc, Blu-ray disc, etc.), magnetic disk storage medium, or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited to these. The memory exists independently and is connected to the processor via a communication bus, or the memory is integrated with the processor.

[0135] A processor may include one or more processing cores. The processor executes instructions, programs, code sets, or instruction sets stored in memory, and calls data stored in memory to perform various functions and process data as described in this application. The processor may be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), controller, microcontroller, and microprocessor. It is understood that, for different devices, the electronic devices used to implement the above-described processor functions may also be other types, and the embodiments of this application do not specifically limit this.

[0136] The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0137] In an alternative embodiment, the computer device may further include a communication interface (not shown) for communication with other devices.

[0138] This application provides a computer-readable storage medium, including, for example, various media capable of storing program code such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. This computer-readable storage medium stores a computer program that can be loaded by a processor and execute the methods of the above embodiments.

[0139] This application also provides a computer program product comprising a computer program tangibly embodied on a readable medium thereof, the computer program containing program code for performing any of the methods described in any of the embodiments of this application, the computer program being downloadable and installable over a network, and / or installed from a removable medium (such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc.).

[0140] The above description of the embodiments is only used to provide a detailed introduction to the technical solutions of this application. However, the description of the above embodiments is only for the purpose of helping to understand the methods and core ideas of this application, and should not be construed as a limitation of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application.

Claims

1. An adaptive deep learning visual odometry localization method for unmanned systems, characterized in that, The method includes: Receive the RGB image to be tested, and for the current frame of the image to be tested, retrieve the previous frame from the history record to form two adjacent RGB images as the positioning input data; The positioning input data is input into the trained positioning model. After the spatiotemporal features are extracted by the adaptive convolution module, the relative pose of the camera is output by the pose estimation network. The localization model is trained in the following manner: RGB images containing real pose labels are received sequentially, and two adjacent RGB images are selected from the received images as training input data. The training input data is subjected to data augmentation processing, which includes motion blur enhancement and basic data augmentation. The motion blur enhancement is used to simulate camera motion artifacts, and the basic data augmentation is used to simulate changes in environmental conditions and occlusion. A localization model containing an adaptive convolutional module and a pose estimation network is constructed. The adaptive convolutional module generates calibration weights by combining the spatial and temporal contexts of adjacent frames, and dynamically adjusts the convolutional kernel weights to extract spatiotemporal features from the data-enhanced image. Based on the aforementioned spatiotemporal features, a pose estimation network outputs a predicted pose, and the mean square error between the true pose and the predicted pose is used as the loss function to iteratively optimize the localization model parameters until the loss function converges. The motion blur includes radial blur and / or directional blur; The radial blur is caused by the translational motion of the camera along the roll axis. Specifically, it is achieved by determining the center point of the image, causing the pixels to shift according to their distance from the center point, and mixing inward and outward blurred sub-images. The directional blur is caused by a combination of translation and rotation of the camera along the yaw and pitch axes. Specifically, it is achieved by simulating with a Gaussian kernel and mixing the pixel with its neighboring pixels in the opposite direction of the motion to produce a trailing effect. The radial blurring process involves generating a sub-image by calculating the pixel value of the target position in the displacement map through linear interpolation, combining the inward and outward blurred sub-images using weights, and repeating the process a preset number of iterations to produce a radial blurring effect. The directional blurring is achieved by applying an affine transformation to the Gaussian kernel using a rotation matrix, aligning the blurred direction with the motion direction. The rotation matrix is ​​calculated based on the Euler angles corresponding to the camera motion. The kernel weight adjustment method of the adaptive convolution module includes: The convolution kernel weight W at time t t The baseline weight Wᵦ and the calibration weight α t The product of; wherein the calibration weight α t The description vector is obtained by performing global average pooling on the input image, and then generated by stacked one-dimensional convolution.

2. The method as described in claim 1, characterized in that, The basic data augmentation includes color jitter and random erasure; the color jitter involves randomly perturbing the color attributes of the image during training to simulate changes in lighting or environmental conditions in the real world; the random erasure simulates occlusion by randomly covering rectangular areas in the image to simulate the situation where objects are partially occluded in real-world scenes.

3. The method as described in claim 2, characterized in that, The color attributes include at least one of brightness, contrast, and saturation.

4. The method as described in claim 1, characterized in that, The pose estimation network is based on the Transformer architecture. It divides the feature map after adaptive convolution into image blocks, combines them with position encoding, processes them through a Transformer encoder, and then outputs a 6-DOF relative pose through a multilayer perceptron.

5. The method as described in claim 1, characterized in that, The loss function includes: in, This represents the batch size (set to 4 in this study). Represents the true relative pose. This indicates the predicted relative pose.

6. An adaptive deep learning visual odometry localization system for unmanned systems, characterized in that, include: The receiving module is used to receive the RGB image to be tested. For the current image frame to be tested, it retrieves the previous frame image from the historical records and forms two adjacent RGB images as positioning input data. The localization model is used to receive the localization input data, extract spatiotemporal features through an adaptive convolution module, and output the relative pose of the camera by a pose estimation network. The localization model is trained in the following manner: The system sequentially receives RGB images containing real pose labels, and selects two adjacent RGB images from the received images as training input data. Data augmentation processing is performed on the training input data, including motion blur enhancement and basic data augmentation. Motion blur enhancement is used to simulate camera motion artifacts, and basic data augmentation is used to simulate changes in environmental conditions and occlusion. A localization model containing an adaptive convolutional module and a pose estimation network is constructed. The adaptive convolutional module combines the spatial and temporal contexts of adjacent frames to generate calibration weights, dynamically adjusting the convolutional kernel weights to extract spatiotemporal features from the data-augmented images. Based on the aforementioned spatiotemporal features, a pose estimation network outputs a predicted pose, and the mean square error between the true pose and the predicted pose is used as the loss function to iteratively optimize the localization model parameters until the loss function converges. The motion blur includes radial blur and / or directional blur; The radial blur is caused by the translational motion of the camera along the roll axis. Specifically, it is achieved by determining the center point of the image, causing the pixels to shift according to their distance from the center point, and mixing inward and outward blurred sub-images. The directional blur is caused by a combination of translation and rotation of the camera along the yaw and pitch axes. Specifically, it is achieved by simulating with a Gaussian kernel and mixing the pixel with its neighboring pixels in the opposite direction of the motion to produce a trailing effect. The radial blurring process involves generating a sub-image by calculating the pixel value of the target position in the displacement map through linear interpolation, combining the inward and outward blurred sub-images using weights, and repeating the process a preset number of iterations to produce a radial blurring effect. The directional blurring is achieved by applying an affine transformation to the Gaussian kernel using a rotation matrix, aligning the blurred direction with the motion direction. The rotation matrix is ​​calculated based on the Euler angles corresponding to the camera motion. The kernel weight adjustment method of the adaptive convolution module includes: The convolution kernel weight W at time t t The baseline weight Wᵦ and the calibration weight α t The product of; wherein the calibration weight α t The description vector is obtained by performing global average pooling on the input image, and then generated by stacked one-dimensional convolution.

Images