Visual positioning method and device based on lifelong learning
By introducing a 'fast learning-continuous memory' dual network framework and experience playback pool in visual odometers, the difficulty of tracking feature points in traditional visual positioning in environments of sparse textures and illumination changes and the problem of 'catastrophic forgetting' in deep learning, Achieve more stable visual positioning performance.
Patent Information
- Application Number
- CN202411250925.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-06
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-09-06
AI Technical Summary
In environments with sparse textures and severe lighting changes, traditional visual simultaneous positioning and mapping systems are difficult to maintain reliable feature point tracking, resulting in failed positioning, and deep learning methods will experience 'catastrophic forgetting' when training new distributed data, resulting in previous Task performance deteriorates.
Using a lifelong learning-based visual positioning method, simulates complementary learning of the hippocampus and cerebral cortex through a 'fast learning-continuous memory' dual network framework and experience replay pool, uses knowledge distillation and consistency error to constrain network parameters, mitigate the catastrophic Forgotten' phenomenon.
It effectively slows down the performance degradation of deep learning models on new tasks, ensures performance retention of previous tasks, and improves the robustness and consistency of visual positioning.
Smart Images

Figure CN118781193B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical fields of computer vision and image processing, and in particular to a visual positioning method and device based on lifelong learning. Background Art
[0002] Robust and high-precision positioning and mapping are the basis for autonomous navigation of unmanned platforms. Among them, the traditional visual simultaneous localization and mapping (SLAM) system based on geometric models has been widely used due to its low cost and lightweight structure.
[0003] The traditional visual simultaneous localization and mapping system based on geometric models is mainly composed of three modules: feature tracking, local map management and closed-loop correction. Among them, the feature tracking and local map management modules solve and optimize the pose according to the matching information of the previous and next frame images and the local map information in the sliding window. These two modules can be collectively referred to as visual odometers; then the closed-loop detection module corrects the accumulated error of the odometer according to the closed-loop nodes detected in the current road section.
[0004] However, in an environment with sparse textures and drastic changes in illumination, it is difficult for the front-end odometer to find enough reliable feature points, which can easily lead to tracking failures. This phenomenon is attributed to the fact that the front-end hand-designed local feature detection and matching algorithm is developed based on low-level image information such as pixel gradients of image blocks. Therefore, drastic changes in illumination can easily affect the performance of the feature point extraction algorithm and cause positioning failure of the overall SLAM system.
[0005] Therefore, a self-supervised monocular visual positioning method based on deep learning (visual odometer based on self-supervised learning) has emerged. This method uses a convolutional neural network to form a visual feature extractor to extract the global features of the image and convert it into pose estimation values end-to-end. It uses a multi-view geometry model to construct an error function to optimize the network training. Through multiple rounds of iterative training, the optimized pose estimation value is obtained. No pose labels are required during the training process. However, this method cannot accurately estimate the scale information of the scene, so its application scenarios are limited.
[0006] Depth estimation is a technology that predicts scene depth information from a single or multiple images. It is a very popular research direction in the field of computer vision and plays a key role in tasks such as 3D reconstruction, scene understanding, and environmental perception. According to different training methods, existing methods are mainly divided into three categories: supervised learning, unsupervised learning, and semi-supervised learning. Supervised methods require a large number of input image and depth map data pairs to drive model training, but due to the difficulty in obtaining real depth maps, such methods are costly. Unsupervised methods do not require data with their own labels to complete model training. The algorithm constructs labels based on the structure and characteristics of the data itself through geometric relationships. This method can make full use of the geometric information in the data to train the model. Using the depth estimation method, depth labels with consistent global scale can be generated.
[0007] The deep learning method is to map the original data to a high-dimensional feature space through the forward propagation of a multi-layer neural network, and convert the original information into high-dimensional implicit features. During the iterative training process, the implicit features will selectively retain the information in the original data that is related to the current task, and the information that is not related to the current task will be discarded. Similarly, the parameters of the deep learning neural network will also be adjusted according to the data distribution of the current task. If a deep learning neural network trained on previous task data is further trained on new task data, the network will iteratively adjust the network parameters in order to reduce the error so that the parameters can fit the data distribution of the new task, and the network parameter information applied to the original task will be lost in large quantities. If there is a significant difference between the two data distributions, the network parameters adjusted on the new task data cannot be applied to the previous task data.
[0008] Visual odometry based on self-supervised learning is essentially a positioning method based on deep learning. Deep learning methods fit the nonlinear relationship between input data and predicted labels on the training set. However, when training newly distributed data, the test performance of the original training data will be greatly reduced. This is called "catastrophic forgetting" in the field of deep learning. Summary of the invention
[0009] Based on this, it is necessary to provide a visual positioning method and device based on lifelong learning to address the above-mentioned technical problems, which can alleviate the phenomenon of "catastrophic forgetting".
[0010] Visual positioning methods based on lifelong learning include:
[0011] Acquire an image sequence, wherein the image sequence includes a plurality of groups of images that are continuous in time, and each group of images includes a target image and a source image;
[0012] Based on the depth estimation network and the pose estimation network, a fast learning model and a persistent memory model are constructed respectively;
[0013] According to the fast learning model and the persistent memory model, reprojected images are obtained respectively to calculate the consistency error; the image sequence is used as an input image, and the fast learning model is trained according to the consistency error to obtain fast learning model parameters;
[0014] The persistent memory model is updated according to the fast learning model parameters to obtain the persistent memory model parameters; the persistent memory model parameters are:
[0015] ;
[0016] In the formula, To persist the model parameters, To estimate network parameters in depth in persistent memory models, To estimate the network parameters of the pose in the persistent memory model;
[0017] Visual localization is performed based on the fast learning model parameters and the persistent memory model parameters.
[0018] In one embodiment, the reprojected images are obtained according to the fast learning model and the persistent memory model, respectively, including:
[0019] Predicting the relative pose between the target image and the corresponding source image according to the fast learning model, warping the source image to the corresponding target image by combining the relative pose with the camera matrix, and obtaining a reprojected image of the fast learning model;
[0020] According to the persistent memory model, the relative pose between the target image and the corresponding source image is predicted, and the source image is warped to the corresponding target image by combining the relative pose with the camera matrix to obtain a reprojected image of the persistent memory model.
[0021] In one embodiment, calculating the consistency error includes:
[0022] The pixel-wise spatiotemporal consistency between the reprojected image of the fast learning model and the reprojected image of the persistent memory model is calculated, and the average of all pixel-wise spatiotemporal consistency is taken as the consistency error.
[0023] In one embodiment, calculating the pixel spatiotemporal consistency between the reprojected image of the fast learning model and the reprojected image of the persistent memory model includes:
[0024] ;
[0025] In the formula, is the pixel temporal and spatial consistency, is the balance parameter, for and The structural similarity index between is the reprojected image of the persistent memory model, Reprojected images for fast learning models.
[0026] In one embodiment, the image sequence is used as an input image, and a fast learning model is trained according to the consistency error to obtain fast learning model parameters, including:
[0027] Taking the image sequence as input images, calculating photometric loss and smoothing loss to constitute a total training loss for unsupervised depth estimation, and obtaining an unsupervised error;
[0028] The fast learning model is trained according to the unsupervised error and the consistency error to obtain fast learning model parameters.
[0029] In one embodiment, the fast learning model parameters are:
[0030] ;
[0031] In the formula, To quickly learn model parameters, To quickly learn the depth estimation network parameters in the model, To quickly learn the pose estimation network parameters in the model.
[0032] In one embodiment, the persistent memory model is updated according to the fast learning model parameters to obtain the persistent memory model parameters, including:
[0033] By means of knowledge distillation learning, the fast learning model parameters are solidified into the own network parameters of the persistent memory model, so as to update the persistent memory model and obtain the persistent memory model parameters.
[0034] In one embodiment, after acquiring the image sequence, the method further includes: constructing an experience replay pool, using the output of the experience replay pool and the image sequence as input images;
[0035] The experience replay pool is used for memory and replay of image sequences.
[0036] Visual positioning device based on lifelong learning, including:
[0037] An acquisition module, used for acquiring an image sequence, wherein the image sequence includes a plurality of groups of images that are continuous in time, and each group of images includes a target image and a source image;
[0038] A construction module for constructing a fast learning model and a persistent memory model based on a depth estimation network and a pose estimation network, respectively;
[0039] A learning module is used to obtain reprojected images according to a fast learning model and a persistent memory model respectively to calculate a consistency error; using the image sequence as an input image, training the fast learning model according to the consistency error to obtain fast learning model parameters;
[0040] The memory module is used to update the persistent memory model according to the fast learning model parameters to obtain the persistent memory model parameters; the persistent memory model parameters are:
[0041] ;
[0042] In the formula, To persist the model parameters, To estimate network parameters in depth in persistent memory models, To estimate the network parameters of the pose in the persistent memory model;
[0043] The positioning module is used to perform visual positioning based on the fast learning model parameters and the continuous memory model parameters.
[0044] The above-mentioned visual positioning method and device based on lifelong learning introduces the complementary learning theory in brain neuroscience into the construction of visual odometry. With reference to the "hippocampus-cortex" complementary learning model, a dual-network visual odometry framework of "fast learning-persistent memory" is designed to form a lifelong learning method. With reference to the recall mechanism in the human brain, an experience replay framework based on pond sampling is designed, which can slow down the "catastrophic forgetting" phenomenon of deep learning and ensure that the network retains the performance of previous tasks. Specifically: the fast learning model simulates the fast learning function of the hippocampus, uses typical self-supervised errors to train the network, and uses the training error gradient to optimize the fast learning model parameters; at the same time, the persistent memory model solidifies the fast learning model parameters into its own network parameters through distillation learning, and constructs a consistency error to constrain the parameter changes of the "persistent memory" network and the "fast learning" network, so as to achieve the purpose of slowing down the "catastrophic forgetting" phenomenon. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 This is a diagram of an application scenario of a visual positioning method based on lifelong learning in one embodiment;
[0046] Figure 2 is a flow chart of a visual positioning method based on lifelong learning in one embodiment;
[0047] Figure 3 is a schematic diagram of the architecture of a visual positioning method based on lifelong learning in one embodiment;
[0048] Figure 4 is a structural block diagram of a visual positioning device based on lifelong learning in one embodiment;
[0049] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0050] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.
[0051] It should be noted that all directional indications in the embodiments of the present application (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components in a certain specific posture. If the specific posture changes, the directional indication will also change accordingly.
[0052] In addition, the descriptions of "first", "second", etc. in this application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "multiple groups" means at least two groups, such as two groups, three groups, etc., unless otherwise clearly and specifically defined.
[0053] In this application, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection, a physical connection, or a wireless communication connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.
[0054] In addition, the technical solutions between the various embodiments of the present application can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.
[0055] The visual positioning method based on lifelong learning provided in this application can be applied to Figure 1In the application scenario diagram shown, the terminal 102 communicates with the server 104 through the network, and the terminal 102 may include but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices, and the server 104 may be various portal websites, servers corresponding to the backend of the work system, etc.
[0056] This application provides a visual positioning method based on lifelong learning, such as Figure 2 The flowchart shown in FIG. 1 shows a method for applying the method to Figure 1 The following is an example of a terminal in the example, including:
[0057] Step 202: Acquire an image sequence, where the image sequence includes multiple groups of images that are continuous in time, and each group of images includes a target image and a source image.
[0058] Specifically: In a group of images, the sequence number of the target image is i, which is expressed as: , the source image is the two frames before and after the target image, expressed as: , where H represents the image height and W represents the image width.
[0059] In this step, after acquiring the image sequence, it also includes: constructing an experience replay pool; the experience replay pool is used for memorizing and replaying the image sequence.
[0060] Specifically: the experience replay pool includes a variational autoencoder and a replay pool. The variational autoencoder includes an encoder and a decoder. The encoder is used to memorize the image sequence, the decoder is used to replay the image sequence, and the replay pool is a Buffeer replay pool.
[0061] Variational Auto Encoder (VAE) is an image compression and restoration framework, which consists of an encoder and a decoder. By inputting an image into the encoder, the image is compressed into a high-dimensional vector and stored in a Buffet playback pool. When the high-dimensional vector needs to be restored to an image, the high-dimensional vector is input into the decoder to restore the image.
[0062] The buffer replay pool is used to memorize and replay the input target image and source image. The buffer replay pool will maintain a certain number of "memories" and randomly select memories to replay during training and participate in training together with the new input image sequence. In order to maintain the largest possible memory space while reducing memory, the original image is not directly memorized during memorization, but VAE is used to compress the image into a high-dimensional vector for storage. When recalling, VAE is used to convert the stored high-dimensional vector back into an image sequence. The size of the buffer replay pool is fixed, and a certain amount of training data is extracted from it with equal probability as replay data and incorporated into the current training data. The design of the replay pool allows the network to review the training tasks of previous tasks while training the current task data, which ensures that the network retains the performance of the previous task.
[0063] After building the experience replay pool, the pre-trained parameters are loaded into the encoder and decoder for subsequent image compression and restoration to complete the initialization of the variational autoencoder.
[0064] Step 204: construct a fast learning model and a persistent memory model based on the depth estimation network and the pose estimation network, respectively.
[0065] Specifically:
[0066] Based on the depth estimation network (DepthNet) and the pose estimation network (PoseNet), a fast learning model is constructed. The fast learning model is trained using the images of the current task and the image data in the playback pool.
[0067] Based on the depth estimation network (DepthNet) and the pose estimation network (PoseNet), a persistent memory model is constructed. The persistent memory model simulates the knowledge solidification function in the cerebral neocortex and solidifies the experiential knowledge learned by the rapid learning model into its own parameters.
[0068] In this step, the depth estimation network (DepthNet) is responsible for the depth estimation of the input image, and the pose estimation network (PoseNet) is responsible for estimating the relative pose of adjacent frames (the target image and the previous frame source image are adjacent frames, and the target image and the next frame source image are also adjacent frames). At the same time, the pose estimation network uses the channel attention model "ECANet" to model the relationship between feature channels in the image visual features, and outputs an attention mask with the same dimension as the original visual features to filter the visual features.
[0069] Step 206, obtaining reprojected images according to the fast learning model and the persistent memory model respectively to calculate the consistency error; using the image sequence as the input image, training the fast learning model according to the consistency error to obtain the fast learning model parameters.
[0070] Specifically:
[0071] According to the fast learning model, the relative pose between the target image and the corresponding source image is predicted, and the source image is warped to the corresponding target image by combining the relative pose and the camera matrix to obtain the reprojected image of the fast learning model;
[0072] According to the persistent memory model, the relative pose between the target image and the corresponding source image is predicted, and the source image is warped to the corresponding target image by combining the relative pose and the camera matrix to obtain the reprojected image of the persistent memory model;
[0073] Calculate the pixel spatiotemporal consistency between the reprojected image of the fast learning model and the reprojected image of the persistent memory model, and take the average of all pixel spatiotemporal consistency as the consistency error;
[0074] Taking the image sequence as the input image, calculate the photometric loss and smoothness loss to constitute the total training loss of unsupervised depth estimation and get the unsupervised error;
[0075] The fast learning model is trained according to the unsupervised error and the consistency error to obtain the fast learning model parameters.
[0076] More specifically:
[0077] For the fast learning model and the persistent memory model, The parameterized DepthNet network predicts inverse depth at four resolutions, which are then bilinearly upsampled to the input resolution to reduce texture copy artifacts. The parameterized PoseNet network predicts the relative pose between each source-target image pair concatenated along the channel dimension, combines this relative pose with the camera's intrinsic matrix, and uses the perspective projection equation to transform each source image Warp to the target image. This process (view synthesis) is performed for each upsampled target depth prediction indexed to obtain a reprojected image, i.e., a synthesized target image .
[0078] Ideally, the fast learning model should have a mechanism to retain prior knowledge, that is, learn in the context of consolidating representations; in addition, if the fast learning model experiences catastrophic forgetting, it will also have a negative impact on the persistent memory model. Therefore, the consolidated representation knowledge of the memory samples is refined from the persistent memory model back to the fast learning model; since depth estimation involves training through view synthesis, it is necessary to ensure the consistency of the synthesis targets between the persistent and fast learning models, which guarantees the spatial consistency in the merged depth map of the target image and the temporal consistency of the pose between the target and nearby source images, that is, the spatiotemporal consistency. That is, in each training step, a batch is sampled from the current task stream , and sample a batch from the memory buffer , calculate the consistency error between these synthetic targets.
[0079] Compute the pixel-wise spatiotemporal consistency between the reprojected image from the fast learning model and the reprojected image from the persistent memory model:
[0080] ;
[0081] In the formula, is the pixel spatiotemporal consistency, and each pixel consistency is of shape The image shape loss, is the balance parameter, for and The structural similarity index between is the reprojected image of the persistent memory model (i.e., using the persistent memory model to The source images of the corresponding left and right frames synthesized by the target images), is the reprojected image of the fast learning model (i.e., using the fast learning model to The source images of the corresponding left and right frames are synthesized from the target images).
[0082] In order to reduce overfitting of the buffered samples, improve invariance through enhancement, and improve efficiency, we need to randomly crop these pixel consistencies before evaluating the final spatiotemporal consistency loss, and draw the proportion of each consistency to be cropped from a Gaussian distribution with a mean of 0.5 and a standard deviation of 0.1. These cropped losses are then averaged over all predictions, source images, and cropped pixels to obtain The average value of the spatiotemporal consistency of all pixels is taken as the consistency error of the target image.
[0083] Take the image sequence as the input image and calculate the photometric loss; specifically: the synthetic image forms an appearance-based photometric loss with the source image to offset the influence of temporarily stationary pixels; an appearance-based photometric error is formed between each synthetic target image and the original target image; the photometric loss connects the depth and pose estimation networks, forming the basis of unsupervised deep learning; in order to offset the influence of temporarily stationary pixels (for example, when there is object motion but no self-motion), the photometric loss is only considered at the pixel positions where the photometric loss is lower than the photometric loss between the undistorted source image and the target image at each scale. This process is called automatic masking. Calculate the smoothness loss and use the pixel-by-pixel edge-aware smoothness loss to regularize the depth prediction; the masked photometric loss and the smoothness loss together constitute the total training loss for unsupervised depth estimation, which is used Represents; Based on the total training loss, the unsupervised error is obtained.
[0084] The fast learning model is trained according to the unsupervised error and the consistency error to obtain the fast learning model parameters.
[0085] The fast learning model is , the fast learning model parameters are:
[0086] ;
[0087] In the formula, To quickly learn model parameters, To quickly learn the depth estimation network parameters in the model, To quickly learn the pose estimation network parameters in the model.
[0088] In this step, when the experience replay pool is constructed, the output of the experience replay pool and the image sequence are used as input images.
[0089] Step 208, updating the persistent memory model according to the fast learning model parameters to obtain the persistent memory model parameters.
[0090] Specifically:
[0091] Through knowledge distillation learning, the fast learning model parameters are solidified into the persistent memory model's own network parameters to update the persistent memory model and obtain the persistent memory model parameters.
[0092] The persistent memory model is , the parameters of the persistent memory model are:
[0093] ;
[0094] In the formula, To persist the model parameters, To estimate network parameters in depth in persistent memory models, Parameters of the pose estimation network in the persistent memory model.
[0095] In this step, knowledge distillation learning is the prior art.
[0096] Step 210, performing visual positioning according to the fast learning model parameters and the continuous memory model parameters.
[0097] In this step, visual positioning includes: multimodal visual odometer and depth estimation. How to perform visual positioning based on fast learning model parameters and persistent memory model parameters belongs to the prior art and will not be described in detail here.
[0098] In this embodiment, if Figure 3As shown in the figure, both the fast learning model and the persistent memory model use a dual network framework of a pose estimation network PoseNet and a depth estimation network DepthNet to calculate the pose and depth estimation values of the fast learning model and the persistent memory model, output two estimation values of pose and depth, and use consistency error to constrain the fast learning model so that the estimation values of the two are relatively consistent.
[0099] The fast learning model is learnable and is updated using ordinary error gradient back propagation, and the weighted sliding average algorithm is used to update the parameters of the persistent memory model. During this process, only the fast learning model transmits information to the persistent memory model, and the persistent memory model also needs to constrain the training of the fast learning model. The information of previous training data stored in the parameters of the persistent memory model is used to optimize the error function of the fast learning model, so that the persistent memory model is maintained as the exponential moving average (EMA) of the fast learning model, thereby realizing the interaction of EMA-based training information between the fast learning model and the persistent memory model.
[0100] Among them, prohibiting gradient updates means: "freezing" the network parameters of the persistent memory model when training the fast learning model, and not allowing the network to perform gradient updates, which means that the weights of these networks remain unchanged during the back propagation process and will not be updated due to changes in the loss function. Specifically: after the persistent memory model and the fast learning model build a consistent loss, the network parameters of the persistent memory model are frozen, and only the consistent loss is allowed to update the fast learning model, which can ensure that the parameters of the persistent memory model will not change while the fast learning model retains the network prediction ability of the persistent memory model.
[0101] In addition, the normal vector label refers to the label of the normal vector of each point or pixel when the model is trained for normal vector estimation. These labels are used to supervise the learning process so that the model can learn to predict the correct normal vector. The normal vector describes the direction of the plane in space. The deep label generated by the network with zero-sample generalization ability can more accurately describe the approximate direction of the plane in the scene. And because the network uses a large number of training sets with deep true value labels, the network can more accurately describe the relative distance relationship of objects in the scene. When calculating the unsupervised error, the convergence speed of the DepthNet network is slow. Therefore, the prior information provided by the deep normal vector label is added to the process of calculating the unsupervised error to constrain the training of the DepthNet network, which can speed up the convergence of the network.
[0102] The above-mentioned visual positioning method based on lifelong learning introduces the complementary learning theory in brain neuroscience into the construction of visual odometry. Referring to the "hippocampus-cortex" complementary learning model, a dual-network visual odometry framework of "fast learning-persistent memory" is designed to form a lifelong learning method. Referring to the recall mechanism in the human brain, an experience replay framework based on pond sampling is designed, which can slow down the "catastrophic forgetting" phenomenon of deep learning and ensure that the network retains the performance of previous tasks. Specifically: the fast learning model simulates the fast learning function of the hippocampus, uses typical self-supervised errors to train the network, and uses the training error gradient to optimize the fast learning model parameters; at the same time, the persistent memory model solidifies the fast learning model parameters into its own network parameters through distillation learning, and constructs a consistency error to constrain the parameter changes of the "persistent memory" network and the "fast learning" network, so as to achieve the purpose of slowing down the "catastrophic forgetting" phenomenon.
[0103] It should be understood that although Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.
[0104] The present application also provides a visual positioning device based on lifelong learning, such as Figure 4 As shown, in one embodiment, it includes: an acquisition module 402, a construction module 404, a learning module 406, a memory module 408 and a positioning module 410, wherein:
[0105] An acquisition module 402 is used to acquire an image sequence, where the image sequence includes a plurality of groups of images that are continuous in time, and each group of images includes a target image and a source image;
[0106] A construction module 404 is used to construct a fast learning model and a persistent memory model according to the depth estimation network and the pose estimation network respectively;
[0107] The learning module 406 is used to obtain reprojected images according to the fast learning model and the persistent memory model respectively to calculate the consistency error; take the image sequence as the input image, train the fast learning model according to the consistency error, and obtain the fast learning model parameters;
[0108] The memory module 408 is used to update the continuous memory model according to the fast learning model parameters to obtain the continuous memory model parameters; the continuous memory model parameters are:
[0109] ;
[0110] In the formula, To persist the model parameters, To estimate network parameters in depth in persistent memory models, To estimate the network parameters of the pose in the persistent memory model;
[0111] The positioning module 410 is used to perform visual positioning according to the fast learning model parameters and the continuous memory model parameters.
[0112] For the specific definition of the visual positioning device based on lifelong learning, please refer to the definition of the visual positioning method based on lifelong learning above, which will not be repeated here. Each module in the above device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0113] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a visual positioning method based on lifelong learning is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.
[0114] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0115] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in the above embodiment when executing the computer program.
[0116] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiment are implemented.
[0117] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0118] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0119] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A visual positioning method based on lifelong learning, characterized in that: include: Acquire an image sequence, wherein the image sequence includes a plurality of groups of images that are continuous in time, and each group of images includes a target image and a source image; Based on the depth estimation network and the pose estimation network, a fast learning model and a persistent memory model are constructed respectively; According to the fast learning model and the persistent memory model, reprojected images are obtained respectively to calculate the consistency error; the image sequence is used as an input image, and the fast learning model is trained according to the consistency error to obtain fast learning model parameters; According to the fast learning model and the persistent memory model, respectively, reprojected images are obtained, including: according to the fast learning model, predicting the relative pose between the target image and the corresponding source image, combining the relative pose with the camera matrix, warping the source image to the corresponding target image, and obtaining the reprojected image of the fast learning model; according to the persistent memory model, predicting the relative pose between the target image and the corresponding source image, combining the relative pose with the camera matrix, warping the source image to the corresponding target image, and obtaining the reprojected image of the persistent memory model; According to the fast learning model parameters, the persistent memory model is updated to obtain the persistent memory model parameters; according to the fast learning model parameters, the persistent memory model is updated to obtain the persistent memory model parameters, including: by means of knowledge distillation learning, the fast learning model parameters are solidified into the own network parameters of the persistent memory model to update the persistent memory model to obtain the persistent memory model parameters; the persistent memory model parameters are: In the formula, To persist the model parameters, To estimate network parameters in depth in persistent memory models, To estimate the network parameters of the pose in the persistent memory model; Visual localization is performed based on the fast learning model parameters and the persistent memory model parameters.
2. The visual positioning method based on lifelong learning according to claim 1, characterized in that: Calculate consistency error, including: The pixel-wise spatiotemporal consistency between the reprojected image of the fast learning model and the reprojected image of the persistent memory model is calculated, and the average of all pixel-wise spatiotemporal consistency is taken as the consistency error.
3. The visual positioning method based on lifelong learning according to claim 2, characterized in that: Compute the pixel-wise spatiotemporal consistency between the reprojected image from the fast learning model and the reprojected image from the persistent memory model, including: In the formula, is the pixel temporal and spatial consistency, is the balance parameter, for and The structural similarity index between is the reprojected image of the persistent memory model, Reprojected images for fast learning models.
4. The visual positioning method based on lifelong learning according to any one of claims 1 to 3, characterized in that: Taking the image sequence as input images and training the fast learning model according to the consistency error to obtain fast learning model parameters, including: Taking the image sequence as input images, calculating photometric loss and smoothing loss to constitute a total training loss for unsupervised depth estimation, and obtaining an unsupervised error; The fast learning model is trained according to the unsupervised error and the consistency error to obtain fast learning model parameters.
5. The visual positioning method based on lifelong learning according to any one of claims 1 to 3, characterized in that: The fast learning model parameters are: In the formula, To quickly learn model parameters, To quickly learn the depth estimation network parameters in the model, To quickly learn the pose estimation network parameters in the model.
6. The visual positioning method based on lifelong learning according to any one of claims 1 to 3, characterized in that: After acquiring the image sequence, the method further includes: constructing an experience replay pool, using the output of the experience replay pool and the image sequence as input images; The experience replay pool is used for memory and replay of image sequences.
7. A visual positioning device based on lifelong learning, characterized in that: include: An acquisition module, used for acquiring an image sequence, wherein the image sequence includes a plurality of groups of images that are continuous in time, and each group of images includes a target image and a source image; A construction module for constructing a fast learning model and a persistent memory model based on a depth estimation network and a pose estimation network, respectively; A learning module is used to obtain reprojected images according to a fast learning model and a persistent memory model respectively to calculate a consistency error; using the image sequence as an input image, training the fast learning model according to the consistency error to obtain fast learning model parameters; According to the fast learning model and the persistent memory model, respectively, reprojected images are obtained, including: according to the fast learning model, predicting the relative pose between the target image and the corresponding source image, combining the relative pose with the camera matrix, warping the source image to the corresponding target image, and obtaining the reprojected image of the fast learning model; according to the persistent memory model, predicting the relative pose between the target image and the corresponding source image, combining the relative pose with the camera matrix, warping the source image to the corresponding target image, and obtaining the reprojected image of the persistent memory model; A memory module is used to update the persistent memory model according to the fast learning model parameters to obtain the persistent memory model parameters; according to the fast learning model parameters, the persistent memory model is updated to obtain the persistent memory model parameters, including: by means of knowledge distillation learning, the fast learning model parameters are solidified into the own network parameters of the persistent memory model to update the persistent memory model to obtain the persistent memory model parameters; the persistent memory model parameters are: In the formula, To persist the model parameters, To estimate network parameters in depth in persistent memory models, To estimate the network parameters of the pose in the persistent memory model; The positioning module is used to perform visual positioning based on the fast learning model parameters and the continuous memory model parameters.
