Self-adaptive deep learning visual mileage positioning method and system for unmanned system

Abstract

The invention relates to the field of robot perception, in particular to a self-adaptive deep learning visual mileage positioning method and system for an unmanned system, and the method comprises the steps: receiving a to-be-detected RGB image, obtaining a previous frame of image from a historical record for a current to-be-detected image frame, and forming two adjacent frames of RGB images as positioning input data; and inputting positioning input data into the trained positioning model, extracting spatial-temporal features through the adaptive convolution module, and outputting the relative pose of the camera through the pose estimation network. The problems that a current visual mileage positioning method is insufficient in generalization ability and poor in positioning precision for complex scenes are solved, the robot and the automatic driving equipment are more accurately and stably positioned in the complex scenes, and the adaptability is higher than that of an existing method.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robot perception and automatic driving precise positioning, in particular to an adaptive deep learning visual odometry positioning method and system for unmanned systems. BACKGROUND

[0002] Visual odometry (VO) is a core technology for autonomous driving and robot navigation, providing real-time positioning information. Visual odometry methods can be broadly classified into two categories: geometric-based and learning-based. Geometric-based methods, such as ORB-SLAM3 and VINS, estimate camera motion by establishing geometric constraints using matching visual features between consecutive frames. These methods are known for their solid theoretical foundation, interpretability, and high accuracy in structured environments, but also have limitations, such as dependence on high-quality feature extraction, sensitivity to matching errors, and insufficient generalization ability in complex or dynamic scenarios.

[0003] In contrast, learning-based VO, such as TartanVO and DeepVO, uses deep neural networks (DNNs) to directly regress camera motion from raw visual input. These methods do not rely on hand-designed geometric models, but instead learn motion states and scene representations autonomously from data. Compared with geometric-based methods, learning-based VO exhibits stronger robustness in challenging conditions (such as low-texture or dynamic scenes) and better generalization ability in new environments after training on diverse datasets. However, such methods often require large amounts of training data and can experience performance degradation in highly dynamic scenarios or unknown environments deviating from the training distribution. SUMMARY

[0004] To solve the problem of insufficient generalization ability and poor positioning accuracy of current visual odometry positioning methods in complex scenarios, the present application provides an adaptive deep learning visual odometry positioning method and system for unmanned systems. Through data augmentation simulating real interference and adaptive algorithms adjusting over time, the model generalization ability is improved, allowing robots and autonomous driving devices to position more accurately and stably in complex scenarios (such as fast movement, light changes, and occlusions), and to be more adaptable than existing methods.

[0005] In a first aspect, the present application provides an adaptive deep learning visual odometry positioning method for unmanned systems:

[0006] Receiving an RGB image to be measured, for the current image frame to be measured, obtaining a previous frame image from historical records to form two adjacent RGB images as positioning input data;

[0007] input the positioning input data into the trained positioning model, and output a relative pose of the camera by a pose estimation network after extracting spatio-temporal features by an adaptive convolution module;

[0008] The positioning model is trained in the following manner:

[0009] RGB images containing real pose labels are sequentially received, and adjacent two frames of the received images are selected as training input data;

[0010] The training input data is subjected to data enhancement processing, and the data enhancement processing includes motion blur enhancement and basic data enhancement, wherein the motion blur enhancement is used to simulate camera motion artifacts, and the basic data enhancement is used to simulate environmental condition changes and occlusions.

[0011] A positioning model including an adaptive convolution module and a pose estimation network is constructed, the adaptive convolution module is used to generate calibration weights by combining spatial context and temporal context of adjacent frames, and the convolution kernel weights are dynamically adjusted to extract spatio-temporal features from the images subjected to data enhancement processing.

[0012] The pose estimation network is used to output a predicted pose based on the spatio-temporal features, and the mean square error between the real pose and the predicted pose is used as a loss function, and the positioning model parameters are iteratively optimized until the loss function converges.

[0013] By adopting the above technical solutions, the sample diversity is expanded by data enhancement during training, so that the model can adapt to complex scenes such as motion blur, illumination change and real dynamic change; and during positioning, adaptive feature extraction is realized to achieve continuous and real-time estimation of camera motion, solving the problems of insufficient generalization ability of the prior art in dynamic scenes and weak inter-frame correlation modeling, and overall improving the positioning accuracy and robustness.

[0014] Optionally, the motion blur includes radial blur and / or directional blur.

[0015] The radial blur is caused by the translational motion of the camera along the roll axis, and is specifically realized by determining a center point of the image, causing the pixel points to be offset and mixed with the sub-images of inward and outward blur according to the distance from the center point.

[0016] The directional blur is caused by the combination of translational and rotational motion of the camera along the yaw axis and the pitch axis, and is specifically realized by simulating a Gaussian kernel and mixing the pixel points with the adjacent pixels in the opposite direction of their motion to produce a trailing effect.

[0017] Optionally, the radial blur includes generating a sub-image by calculating the pixel value of the target position of the displacement map through linear interpolation, and combining the sub-images of inward and outward blur using weights, and repeating the radial blur effect for a predetermined number of iterations.

[0018] Optionally, the direction blur is applied by an affine transformation of the Gaussian kernel through a rotation matrix, which aligns the blur direction with the motion direction, wherein the rotation matrix is calculated based on Euler angles corresponding to the camera motion.

[0019] Optionally, the basic data augmentation includes color jittering and random erasing; the color jittering includes simulating changes in lighting or environmental conditions in the real world by randomly perturbing color attributes of the image during training; the random erasing simulates occlusions by randomly masking rectangular regions in the image, for simulating the case that objects are partially occluded in real scenes.

[0020] Optionally, the color attributes include at least one of brightness, contrast, and saturation.

[0021] Optionally, the adaptive convolution module includes a convolution kernel weight adjustment manner comprising:

[0022] a convolution kernel weight W at time t t is a reference weight W β is multiplied by a calibration weight α t ; wherein the calibration weight α t is obtained by global average pooling of the input image, and then generated by stacked one-dimensional convolution.

[0023] Optionally, the pose estimation network is based on a Transformer architecture, divides the feature map processed by the adaptive convolution into image blocks, combines position encoding, and then processes through a Transformer encoder, and finally outputs a 6-DOF relative pose through a multi-layer perception.

[0024] Optionally, the loss function includes:

[0025]

[0026] wherein B is the batch size (set to 4 in this study), y b,i represents the true relative pose, represents the predicted relative pose.

[0027] In a second aspect, the present application provides an adaptive deep learning visual odometry system for unmanned systems; the system includes various modules for executing the method of the first aspect or any possible implementation manner of the first aspect:

[0028] The system includes:

[0029] The receiving module is configured to receive an RGB image to be measured, obtain a previous frame image from a historical record for a current image frame to be measured, and compose two adjacent RGB images as positioning input data.

[0030] A positioning model is configured to receive the positioning input data, extract spatio-temporal features through an adaptive convolution module, and output a relative pose of the camera by a pose estimation network.

[0031] The positioning model is trained in the following manner:

[0032] RGB images containing real pose labels are sequentially received, and adjacent two frames of the received images are selected as training input data. The training input data is subjected to data enhancement processing, which includes motion blur enhancement and basic data enhancement. The motion blur enhancement is configured to simulate camera motion artifacts, and the basic data enhancement is configured to simulate environmental condition changes and occlusions. A positioning model containing an adaptive convolution module and a pose estimation network is constructed. The adaptive convolution module is configured to generate calibration weights by combining spatial context and temporal context of adjacent frames, and dynamically adjust convolution kernel weights to extract spatio-temporal features from the data-enhanced images. The pose estimation network is configured to output a predicted pose based on the spatio-temporal features. The mean square error between the real pose and the predicted pose is used as a loss function, and the parameters of the positioning model are iteratively optimized until the loss function converges.

[0033] In a third aspect, the present application provides a computer device, comprising a processor, a memory, and a communication bus configured to realize communication connection between the processor and the memory. The processor is configured to execute a computer program stored in the memory to realize the method according to any one of the above aspects.

[0034] In a fourth aspect, the present application further provides a computer readable storage medium storing a computer program. The computer program can be executed by a processor to realize the method according to the above aspects.

[0035] In a fifth aspect, the present application further provides a computer program product comprising a computer program executable by a processor to realize the method according to the above aspects.

[0036] The present application at least has the following beneficial technical effects:

[0037] 1. By simulating data enhancement of real interference and adaptive algorithm adjusting over time, the generalization ability of the model is improved, and the robot and the automatic driving device are more accurate and stable in positioning in complex scenes, and have better adaptability than existing methods. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 A visual odometry positioning method flowchart is provided for the embodiments of the present application.

[0039] Figure 2 A positioning model training flowchart is provided for the embodiments of the present application.

[0040] Figure 3 A visual odometry positioning algorithm architecture diagram provided for an embodiment of the present application;

[0041] Figure 4 An image contrast diagram before and after applying radial blur enhancement provided for an embodiment of the present application;

[0042] Figure 5 An image contrast diagram before and after applying radial blur enhancement (different kernel size) provided for an embodiment of the present application;

[0043] Figure 6 A trajectory diagram of representative sequences estimated by different methods provided for an embodiment of the present application;

[0044] Figure 7 An average error result and variance diagram of test sequences provided for an embodiment of the present application;

[0045] Figure 8 An experimental result diagram of different methods on test sequences of the KITTI dataset provided for an embodiment of the present application;

[0046] Figure 9 An experimental result diagram of each frame pose estimation time of different methods provided for an embodiment of the present application;

[0047] Figure 10 A result comparison diagram of different VO on sequence 09 and sequence 10 provided for an embodiment of the present application;

[0048] Figure 11 A result diagram of the present algorithm using different enhancement and using or not using adaptive convolution provided for an embodiment of the present application;

[0049] Figure 12 An effect diagram of using motion blur enhancement probability provided for an embodiment of the present application;

[0050] Figure 13 An effect diagram of using different radial blur coefficients provided for an embodiment of the present application;

[0051] Figure 14 A visual odometry system structure schematic diagram provided for an embodiment of the present application;

[0052] Figure 15 A computer device structure schematic diagram provided for an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application.

[0054] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to be limiting of the present application. As used in the specification and the appended claims of the present application, the singular forms "a," "an" and "the" are intended to include plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "and / or" as used herein refer to and encompass any or all possible combinations of one or more of the associated listed items. The term "exemplary" means "serving as an example, instance, or illustration," and any embodiment described as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. The terms "first," "second," and the like do not denote any absolute or relative importance, but are used only for descriptive purposes. Thus, a feature with "first," "second" limitation can explicitly or implicitly include one or more such features, and the meaning of "a plurality" is two or more, unless otherwise specified in the description of the embodiments of the present application.

[0055] Reference Figure 1 An adaptive deep learning visual odometry positioning method for unmanned systems mainly includes the following steps:

[0056] S102: receiving an RGB image to be measured, for the current image frame to be measured, obtaining a previous frame image from the historical record, and composing two adjacent frame RGB images as positioning input data;

[0057] S104: inputting the positioning input data into the trained positioning model, extracting the space-time features through the adaptive convolution module, and outputting the relative pose of the camera by the pose estimation network.

[0058] Reference Figure 2 The positioning model is trained in the following manner and mainly includes the following steps:

[0059] S202: sequentially receiving RGB images containing real pose labels, and selecting two adjacent frame RGB images from the received images as training input data;

[0060] S204: performing data enhancement processing on the training input data; the data enhancement processing includes motion blur enhancement and basic data enhancement, wherein the motion blur enhancement is used to simulate camera motion artifacts, and the basic data enhancement is used to simulate environmental condition changes and occlusions;

[0061] Although the conventional way can alleviate the problem of network model data scarcity and insufficient robustness through data augmentation. However, the conventional image enhancement technology (such as flipping and rotating) tends to destroy the spatio-temporal consistency in continuous visual data, resulting in the model being unable to use these training data. To this end, the embodiment simulates motion blur, seasonal changes and occlusion in the training sequence to increase the diversity of data.

[0062] Optionally, the motion blur includes radial blur and / or directional blur. The radial blur is caused by the translational motion of the camera along the roll axis, specifically by determining the center point of the image, causing the pixel points to be offset and mixed according to the distance from the center point to the inner and outer blurred sub-images. The directional blur is caused by the combination of translational and rotational motion of the camera along the yaw and pitch axes, specifically by simulating a Gaussian kernel and mixing the pixel points with their adjacent pixels in the opposite direction of motion to produce a trailing effect.

[0063] Optionally, the radial blur includes generating a sub-image by calculating the pixel value of the target position of the displacement map through linear interpolation, and combining the inner and outer blurred sub-images using weights, and repeating the radial blur effect for a predetermined number of iterations. The directional blur applies an affine transformation to the Gaussian kernel through a rotation matrix, so that the blur direction aligns with the motion direction, wherein the rotation matrix is calculated based on the Euler angles corresponding to the camera motion.

[0064] Optionally, the basic data augmentation includes color jittering and random erasing; the color jittering includes simulating changes in lighting or environmental conditions in the real world by randomly perturbing the color attributes of the image during training; the random erasing simulates the situation where an object is partially occluded in a real scene by randomly masking a rectangular region in the image.

[0065] Optionally, the color attribute includes at least one of brightness, contrast and saturation.

[0066] S206: Construct a positioning model containing an adaptive convolution module and a pose estimation network, generate calibration weights by combining the spatial and temporal contexts of adjacent frames through the adaptive convolution module, and dynamically adjust the convolution kernel weights to extract spatio-temporal features from the data-augmented images;

[0067] The existing odometry method is based on the assumption of spatial translation invariance, and shares the convolution weights between different frames, resulting in the loss of modeling ability in the time dimension. To this end, the embodiment considers the time correlation between frames, dynamically calibrates the weights of the convolution kernel instead of keeping them unchanged, thereby improving the data correlation ability of the network model in context awareness and the accuracy of pose estimation.

[0068] Optionally, the convolution kernel weight adjustment mode of the adaptive convolution module includes:

[0069] Convolution kernel weight W at time tt The benchmark weight W β With calibration weight α t The product of; where the calibration weight α t The description vector is obtained by performing global average pooling on the input image, and then generated by stacked one-dimensional convolution.

[0070] Optionally, the pose estimation network is based on the Transformer architecture, which divides the feature map after adaptive convolution into image blocks, combines them with position encoding, processes them through a Transformer encoder, and then outputs a 6-DOF relative pose through a multilayer perceptron.

[0071] S208: Based on spatiotemporal features, the pose estimation network outputs the predicted pose, and the mean square error between the true pose and the predicted pose is used as the loss function to iteratively optimize the localization model parameters until the loss function converges.

[0072] Optionally, the loss function includes:

[0073]

[0074] Where B is the batch size (set to 4 in this study), y b,i Represents the true relative pose. This indicates the predicted relative pose.

[0075] Optionally, this visual odometry method can be based on, for example, Figure 3 The illustrated algorithm (Data Augmentation and Adaptive Transformer-based VO, RADVO) architecture comprises two main modules: a data augmentation module and an adaptive Transformer-based VO module. The data augmentation module enhances data diversity by simulating rapid camera movement, weather changes, and occlusion. The adaptive Transformer-based VO module estimates the camera pose by combining adaptive convolutions with popular Transformer networks.

[0076] Motion blur enhancement aims to simulate highly dynamic scenes, making models robust to motion blur that occurs in the real world. For example, cameras mounted on robots or vehicles often capture blurred images when moving rapidly. If a model is trained only on sharp images, it may fail when faced with motion blur. In this embodiment, two motion blur enhancement strategies are proposed: radial blur and directional blur.

[0077] Radial blur is typically caused by the translational motion of the camera along the roll axis. This translational motion can be derived from real data. In continuous camera motion, the relative pose transformation T∈SE(3) between two frames. Ndenotes motion, including a rotation matrix R ∈ SE(3) N and a translation vector t ∈ R 3×1 . By adjusting the camera motion speed with different scaling factors, data with different blur levels can be obtained. It is worth noting that the sampling frequency of the camera remains constant during this process.

[0078] The radial blur operator is a technique that shifts image pixels according to their distance from a specified center point, producing a blur effect that radiates outward. To achieve this effect, this embodiment introduces two key parameters: a blur intensity parameter blur, which controls the intensity of the blur effect, as shown in Figure 4 ; and an iteration parameter iterations, which determines the number of times the blur operation is performed. Let the width and height of the image be w and h, respectively, then the coordinates of the image center point are center x = h / 2 and center y = w / 2, and the calculation formula is as follows:

[0079] blur = t roll * η, iterations = C, C ∈ [1, M], (1)

[0080] where t roll represents translation along the roll axis, which can be obtained from the relative transformation matrix T; η is a noise factor randomly generated in the interval [0, 1]; C is a random integer ranging from 1 to M, and M is a constant.

[0081] For a pixel located at (i, j) (where i is the row index and j is the column index), the displacement caused by the blur effect can be decomposed into horizontal and vertical components. The position after displacement is represented as Map[i, j]:

[0082]

[0083] where Δ i and Δ j are the offsets in the row and column directions, respectively. The ± sign determines the direction of the blur: a positive sign indicates outward diffusion, and a negative sign indicates inward contraction. After combining the blur coefficient, the displacement can be further represented as:

[0084] Map[i, j] = [i + Δ i * blur, j + Δ j * blur]. (3)

[0085] By calculating Map[i, j] for all pixel positions, a displacement matrix Map xy ∈ R h×w can be constructed, which encapsulates the displacement of the image in the height and width dimensions, expressed as:

[0086]

[0087] According to the selection of the sign of the offset term Δ i and Δ j two different displacement maps can be generated: representing outward blur, representing inward blur.

[0088] Subsequently, the pixel values of the target locations specified by these displacement maps are calculated using an interpolation method, generating two sub-images. These two sub-images are combined using weights w1and w2:

[0089]

[0090] where the map(·) function applies linear interpolation with weights set to w1= w2= 0.5. This process is repeated iterations times to gradually produce the radial blur effect, expressed as:

[0091]

[0092] Directional blur is usually caused by a combination of translational and rotational motion along the yaw and pitch axes. This embodiment uses a Gaussian kernel to simulate directional blur by blending a pixel with its neighboring pixels in the opposite direction of motion, producing a trailing effect, as shown in Figure 5 The implementation process is as follows:

[0093] First, an initial Gaussian kernel with an odd size is created, with the default blur direction being horizontal. Its expression is:

[0094]

[0095] where i and j are the row and column indices, respectively. Then, to align the blur direction with the motion direction, an affine transformation is applied to the Gaussian kernel through a rotation matrix R. This rotation matrix can be decomposed into Euler angles φ, θ, ψ, and then a two-dimensional rotation matrix for the Gaussian kernel is calculated, with the formula as follows:

[0096]

[0097] To approximate the principal direction of camera motion, the dominant angle can be selected as:

[0098] angle = max(|φ|, |θ|, |ψ|), (9)

[0099] where the absolute value ensures that the largest rotation amplitude is selected, regardless of its positive or negative. Based on this dominant angle, the affine transformation matrix Rot(angle) is calculated:

[0100]

[0101] Finally, the Gaussian kernel is transformed according to this affine transformation matrix:

[0102] kernel' = Wrap(kernel, Rot(angle)), (11)

[0103] where Wrap denotes an affine transformation operation (e.g., warping or resampling the kernel grid). The transformed kernel' reflects the blur direction.

[0104] The present embodiment also considers that the base augmentation includes color jittering and random erasing. Color jittering augmentation simulates the changes in lighting or environmental conditions in the real world by randomly perturbing the color attributes (such as brightness, contrast, and saturation) of the images during training. This forces the model to learn robust features that are invariant to color distortions, thus improving the generalization capability of VO.

[0105] Random erasing is used to simulate the situation where objects are partially occluded in real scenarios (such as obstacles, lighting, or sensor noise). By randomly masking rectangular regions of different sizes, shapes, and positions in the training images, controllable occlusions are introduced. This encourages the VO model to learn robust and diverse feature representations, avoiding over-reliance on specific visual patterns or regions in the data.

[0106] The adaptive Transformer-based VO model includes two modules: an adaptive convolution module and a pose estimation network. Previous learning-based VO works usually use shared network weights across all spatial locations and temporal frames. This can affect the effectiveness of extracting spatio-temporal features, which in turn reduces the accuracy of VO. To this end, the present embodiment introduces adaptive convolution to better capture inter-frame correspondences. Instead of fixing the weights of adaptive convolution kernels across different frames, the present embodiment dynamically adjusts them by multiplying the original weights with a calibration weight t at time t, w t , is decomposed into the product of a baseline weight W b and a calibration weight a t :

[0107] W t = a t x W b (12)

[0108] where the calibration weight a t is adaptively generated according to the input data across all channels of the baseline weight W b . To capture spatio-temporal information, the weight a t needs to consider both the spatial context and the temporal context of neighboring frames. The adaptive convolution module takes augmented images as input and outputs feature maps. For each frame, the input image x tThe global average pooling (GAP) is performed on the spatial dimensions to obtain the descriptor vector v t = GAP(x t ) ∈ R T×C Subsequently, the calibration weights are generated by applying two stacked one-dimensional convolutions. This process can be represented as:

[0109]

[0110] where Conv denotes a convolutional layer, Act is a ReLU activation function, and BN denotes batch normalization. To obtain a broader view, the global descriptor g is incorporated into the weight generation process by a linear mapping function FC, which further integrates the global temporal information. The weights a t can be represented as:

[0111]

[0112] where g is obtained by global average pooling over the temporal and spatial dimensions.

[0113] The pose estimation network is developed based on the TimeSFormer, which takes the feature maps output by the adaptive convolutional module as input and outputs a 6 degree of freedom (6-DoF) pose. Specifically, the input feature maps are first divided into non-overlapping p x p size blocks. Then, each block at time step t (where p ∈ {1,..., N} is vectorized and combined with the learnable position encoding E pos and the classification label cls to form a sequence This sequence is then processed by a Transformer encoder, which adopts a computationally efficient "spatio-temporal" self-attention mechanism to model the intra-frame spatial correlation and inter-frame temporal dependency by decomposing the attention heads. The resulting spatio-temporal feature representation is gradually refined by a Multilayer Perceptron (MLP) projection, ultimately generating an accurate 6-DoF pose estimate. To train the network, the mean squared error (MSE) between the true pose and the predicted pose is used as the loss function:

[0114]

[0115] where B is the batch size (set to 4 in this embodiment), y b,i represents the true 6-DoF relative pose, and y represents the predicted relative pose.

[0116] In this embodiment, the learning rate is set to 0.00001, the training round is 100 rounds, the optimizer uses Adam and uses its default parameters. For the hyperparameters of data enhancement, the enhancement uses a probability of 0.1. For radial blur, the maximum number of iterations is 8.

[0117] Based on the technical solutions provided by the embodiments of the present application, the regularization means of data enhancement is added compared with the existing method, which is a data enhancement method suitable for visual odometry; the motion blur enhancement technology designed for visual odometry simulates challenging scenarios in the real world (such as sudden camera motion), forcing the network to learn feature representations that are invariant to blur, thereby improving the robustness to motion degradation.

[0118] In addition, the adaptive Transformer architecture is combined to explicitly learn the geometric and temporal relationships in visual data, and the adaptive adjustment of convolution weights can better capture spatio-temporal features to cope with environmental dynamics. Therefore, it achieves excellent performance compared with other methods. The effectiveness of the proposed method is evaluated through the public data set KITTI, and the evaluation process has the following comparisons:

[0119] The running results of the method proposed in the embodiments of the present application and the benchmark method on the automobile data set KITTI (international authoritative automatic driving data set) are compared. The benchmark method includes ORB-SLAM3, DeepVO, TSformer-VO, and SWformer-VO. Among them, sequences 00, 02, 08, and 09 are used for training, and the remaining sequences are used for testing. From Figure 6 It can be seen that the method of the embodiments of the present application has obvious competitiveness compared with all benchmark methods on multiple sequences. In Figure 7 In the above table, in addition to the average relative rotation error being in the second place, the test error of the method of the embodiments of the present application is lower than that of all other benchmark methods.

[0120] The detailed quantitative results of each test sequence compared with the benchmark method are also compared, as shown in Figure 8 It can be seen that RADVO ranks first in almost all indicators, and only slightly lags behind ORB-SLAM3 in the average rotation error.

[0121] The calculation cost of the present application and the benchmark method is also compared. As shown in Figure 9 It can be seen that the present application ranks second in the pose estimation time of each frame, only second to TSformer-VO. RADVO uses the same backbone network as TSformer-VO, but RADVO introduces an adaptive convolution layer, so it takes more time. However, compared with other methods, RADVO still remains competitive.

[0122] The proposed method is compared with popular unsupervised learning VO methods, including SFMlearner, Depth-VO-Feat, GeoNet, Bian et al., Masked GANs and the method of Wang et al. Figure 10 It is shown that the proposed method performs best in terms of average translation error and at a medium level in terms of average rotation error, and is still competitive without considering geometric and scale consistency constraints.

[0123] In this embodiment, an ablation study is also conducted on the proposed method, and the effects of using different enhancement strategies and whether to include an adaptive convolution module are first analyzed. Figure 11 It is shown that each component improves performance to some extent, and the maximum gain is obtained when all components are combined, where MB represents motion blur, AC represents adaptive convolution, and BA represents basic enhancement strategy.

[0124] The use probability of motion blur enhancement is set to a value between 0.1 and 0.5. Figure 12 The results show that as the use probability increases, most performance indicators show a downward trend. This can be explained as too frequent application of data enhancement can seriously disturb the data distribution. Therefore, a higher probability is not always better, and an appropriate value should be selected according to the characteristics of the data set. The degree of motion blur enhancement is determined by the blur coefficient and the number of iterations. In practice, the number of iterations is usually randomized to enhance data diversity. In the experiment, the number of iterations is fixed at 8, and the blur coefficient is tested in the range of [0.001, 0.005]. Figure 13 It is shown that as the coefficient increases to 0.002, the values of most performance indicators first decrease and then generally increase, indicating a degradation in accuracy. From the perspective of visual perception, when the coefficient reaches 0.004, the image is significantly blurred, making it difficult to estimate the pose. Therefore, it is important to choose an appropriate coefficient value to achieve satisfactory performance.

[0125] On the basis of the above method embodiment, the embodiment also provides an adaptive deep learning visual odometry system for unmanned systems, which can be used to implement the steps of the above method.

[0126] Reference Figure 14 The visual odometry system comprises:

[0127] The receiving module 141 is configured to receive an RGB image to be measured, and for a current image frame to be measured, obtain a previous frame image from a historical record to form adjacent two frame RGB images as positioning input data.

[0128] The positioning model 142 is configured to receive the positioning input data, extract spatio-temporal features through an adaptive convolution module, and output a relative pose of the camera by a pose estimation network.

[0129] The positioning model is trained in the following manner: sequentially receiving RGB images containing real pose labels, selecting two adjacent RGB images from the received images as training input data; performing data enhancement processing on the training input data, the data enhancement processing including motion blur enhancement and basic data enhancement, wherein the motion blur enhancement is used to simulate camera motion artifacts, and the basic data enhancement is used to simulate environmental condition changes and occlusions; constructing a positioning model containing an adaptive convolution module and a pose estimation network, generating calibration weights by combining the spatial and temporal contexts of adjacent frames through the adaptive convolution module, dynamically adjusting the convolution kernel weights to extract spatio-temporal features from the data-enhanced images; outputting a predicted pose based on the spatio-temporal features through the pose estimation network, taking the mean square error between the real pose and the predicted pose as the loss function, and iteratively optimizing the positioning model parameters until the loss function converges.

[0130] The various variations and specific examples in the method provided by the above embodiments are also applicable to the system of the present embodiment. By the foregoing detailed description of the method, those skilled in the art can clearly understand the implementation method of the system in the present embodiment. For the sake of brevity of the specification, the implementation method of the system in the present embodiment will not be described in detail here.

[0131] In order to better execute the program of the above method, the present embodiment further provides a computer device, as shown in the accompanying drawings, Figure 15 As shown in the accompanying drawings, the computer device includes a processor 151, a memory 152, and a communication bus 153 for realizing the communication connection between the processor 151 and the memory 152.

[0132] The computer device can be implemented in various forms, including mobile phones, tablets, palmtop computers, notebook computers, and desktop computers, etc.

[0133] The memory can be used to store instructions, programs, codes, code sets or instruction sets. The memory can include a storage program area and a storage data area, wherein the storage program area can store instructions for implementing an operating system, instructions for at least one function, and instructions for implementing the method provided by the above embodiments, etc.; the storage data area can store data involved in the method provided by the above embodiments, etc.

[0134] Optionally, the memory is a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable read-only memory (EEPROM), an optical disk (including a compact disc read-only memory (CD-ROM), a compact disc, a laser disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to the above. The memory exists independently and is connected with the processor through a communication bus, or the memory is integrated with the processor.

[0135] The processor can include one or more processing cores. The processor invokes data stored in the memory by running or executing instructions, programs, code sets or instruction sets stored in the memory, performs various functions and processes data of the present application. The processor can be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller and a microprocessor. It can be understood that for different devices, the electronic devices used to implement the functions of the above processor can also be others, and the embodiments of the present application are not limited specifically.

[0136] The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0137] In an optional embodiment, the computer device can further include a communication interface (not shown) for communication with other devices.

[0138] The embodiments of the present application provide a computer readable storage medium, for example, including: a U disk, a mobile hard disk, a Read Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes. The computer readable storage medium stores a computer program capable of being loaded by a processor and executing the method of the above embodiments.

[0139] The embodiments of the present application also provide a computer program product, which includes a computer program tangibly embodied on a computer readable medium, the computer program containing program codes for executing any of the methods of the embodiments of the present application. The computer program can be downloaded and installed on a network, and / or installed from a removable medium (such as a disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc.).

[0140] The above embodiments are only used to specifically describe the technical solutions of the present application, and the above descriptions of the embodiments are only used to help understand the method and the core idea of the present application, and should not be construed as limiting the present application. Those skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. An adaptive deep learning visual odometry method for unmanned systems, characterized in that, The method comprises: receiving an RGB image to be tested, obtaining a previous frame image from a historical record for a current image frame to be tested, and composing two adjacent frame RGB images as positioning input data; inputting the positioning input data into a trained positioning model, extracting spatio-temporal features through an adaptive convolution module, and outputting a relative pose of the camera by a pose estimation network; The positioning model is trained by the following method: receiving RGB images containing real pose labels in sequence, and selecting two adjacent frame RGB images from the received images as training input data; performing data enhancement processing on the training input data, wherein the data enhancement processing includes motion blur enhancement and basic data enhancement, wherein the motion blur enhancement is used to simulate camera motion artifacts, and the basic data enhancement is used to simulate environmental condition changes and occlusions; constructing a positioning model comprising an adaptive convolution module and a pose estimation network, generating calibration weights by combining the spatial context and the time sequence context of adjacent frames through the adaptive convolution module, and dynamically adjusting the convolution kernel weights to extract spatio-temporal features from the data-enhanced images; outputting a predicted pose based on the spatio-temporal features through the pose estimation network, taking the mean square error between the real pose and the predicted pose as a loss function, and iteratively optimizing the positioning model parameters until the loss function converges.

2. The method of claim 1, wherein, The motion blur includes radial blur and / or directional blur; The radial blur is caused by the translational motion of the camera along the roll axis, specifically by determining the center point of the image, causing the pixel points to shift and mix the sub-images of inward and outward blur according to the distance from the center point; The directional blur is caused by the combination of translational and rotational motion of the camera along the yaw and pitch axes, specifically by simulating a Gaussian kernel and mixing the pixel points with their adjacent pixels in the opposite direction of motion to produce a trailing effect.

3. The method of claim 2, wherein, The radial blur includes calculating the pixel values of the target positions of the displacement map to generate sub-images by linear interpolation, and combining the inward and outward blurred sub-images using weights, and repeating the radial blur effect for a predetermined number of iterations.

4. The method of claim 2 or 3, wherein, The directional blur applies affine transformation to the Gaussian kernel through a rotation matrix to align the blur direction with the motion direction, wherein the rotation matrix is calculated based on the Euler angles corresponding to the camera motion.

5. The method of claim 1, wherein, The basic data enhancement includes color jittering and random erasing; the color jittering includes simulating changes in lighting or environmental conditions in the real world by randomly perturbing the color attributes of the image during training; the random erasing simulates the case where an object is partially occluded in a real scene by randomly masking rectangular regions in the image.

6. The method of claim 5, wherein, The color attributes include at least one of brightness, contrast, and saturation.

7. The method of claim 1, wherein, The convolution kernel weight adjustment method of the adaptive convolution module includes: Convolution kernel weights W at time t t For reference weights W β The product of the calibration weight α t The calibration weight α t The description vector is obtained by global average pooling on the input image, and then generated by stacked one-dimensional convolution.

8. The method of claim 1, wherein, The pose estimation network is based on a Transformer architecture, divides the feature map processed by the adaptive convolution into image blocks, processes them through a Transformer encoder after combining the position encoding, and outputs a 6-DOF relative pose through a multi-layer perception.

9. The method of claim 1, wherein, The loss function includes: where B is the batch size (set to 4 in this study), y b,i denotes the true relative pose, denotes the predicted relative pose.

10. An adaptive deep learning visual odometry system for unmanned systems, comprising: includes: The receiving module is configured to receive an RGB image to be tested, acquire a previous frame image from a history record for a current frame image to be tested, and compose two adjacent frame RGB images as positioning input data; The positioning model is configured to receive the positioning input data, extract spatio-temporal features through an adaptive convolution module, and output a relative pose of a camera through a pose estimation network; The positioning model is obtained through training in the following manner: RGB images containing real pose labels are received in sequence, two adjacent frame RGB images are selected from the received images as training input data, data enhancement processing is performed on the training input data, the data enhancement processing includes motion blur enhancement and basic data enhancement, the motion blur enhancement is used to simulate camera motion artifacts, and the basic data enhancement is used to simulate environmental condition changes and occlusions, a positioning model containing an adaptive convolution module and a pose estimation network is constructed, calibration weights are generated through the adaptive convolution module in combination with spatial context and time sequence context of adjacent frames, and convolution kernel weights are dynamically adjusted to extract spatio-temporal features from the images after the data enhancement processing; A predicted pose is output through the pose estimation network based on the spatio-temporal features, a mean square error between the real pose and the predicted pose is taken as a loss function, and positioning model parameters are iteratively optimized until the loss function converges.

Images