A robot scene recognition method based on lifelong learning

Abstract

The present invention discloses a robot scene recognition method based on lifelong learning, which improves the robot's scene recognition accuracy and lifelong learning ability by integrating advanced feature extraction technology and aggregation technology, combining long-term and short-term memory mechanisms, and intelligent knowledge distillation and memory-aware synaptic strategies. The lifelong learning robot scene recognition method of the present invention has significant performance improvements, especially in complex and changing environments. Compared with the existing technology, the method can provide higher recognition accuracy and better long-term learning ability. Through application and testing on Nordland datasets in multiple seasons and Oxford datasets in multiple lighting conditions, the present invention demonstrates its powerful recognition ability for different scenes, as well as its ability to balance the retention of old knowledge and the learning of new knowledge in lifelong learning scenarios.

Description

Technical Field

[0001] The invention belongs to the technical field of robots, and in particular relates to a robot scene recognition method based on lifelong learning. Background Art

[0002] In a broad sense, loop detection is equivalent to scene recognition, and the goal is to find the current scene. Loop detection is an important part of modern simultaneous localization and mapping (SLAM) systems. It is mainly used to eliminate the accumulated offset error after loop detection, improve the accuracy of mapping, and realize the scene recognition function of the robot. The goal of loop detection is to be able to recognize the environment if it has been through it when given an image of the environment, so that the robot has the ability to recognize the scene.

[0003] In the past, traditional loop detection methods used manually calculated features, such as HoG, SIFT, etc. to generate visual words and build a bag-of-words (BoW) model. Later, with the development of artificial intelligence, people began to use deep learning methods to implement loop detection. The main model consists of two parts: a feature extractor responsible for extracting several features in the image and an aggregator that integrates multiple features. Finally, the results output by the aggregator are compared to convert the loop detection problem into an image retrieval problem.

[0004] Although such a method can theoretically realize the scene recognition of robots, it will encounter many other problems in practical applications. The robots actually deployed in the project will not only be active in a single scene, so the scene recognition ability of the robot needs to be able to cope with a variety of dynamically increasing environments, and the storage capacity of the robot is limited. If we do not design it specially, there will only be two results: 1. The robot forgets the recognition ability of the old scene after learning the new scene, which is catastrophic forgetting. 2. The robot retains the data of all scenes, but as the robot is used, the storage overhead will become huge, making it difficult to achieve online deployment. Therefore, in order to solve the above problems, the present invention proposes a robot scene recognition method based on lifelong learning.

[0005] Lifelong learning is a learning process that imitates humans. Humans are not isolated in the learning process, but constantly accumulate past knowledge and use past knowledge to learn more knowledge. Although there will be a certain degree of forgetfulness, as time goes by, humans will learn more and more knowledge. Lifelong learning has three characteristics: 1. Continuous learning. 2. Knowledge is accumulated in the knowledge base. 3. Using past knowledge can help learn future knowledge. Applying lifelong learning to robot scene recognition can help robots learn more scenes without causing catastrophic forgetting.

[0006] In recent years, some methods based on lifelong learning have also achieved robot scene recognition, but they have defects in the following aspects: first, the simple pooling method used in the aggregator cannot adapt well to continuously captured scene images; then, the triple loss function is usually used in contrastive learning, which leads to an unstable training process and poor learning efficiency; secondly, the design of knowledge distillation is too simple and does not take into account the overall distillation effect of the model; finally, the design of the memory module in memory replay is not detailed enough, resulting in poor performance.

[0007] The present invention designs a loop detection method for robot scene recognition based on lifelong learning. The lifelong learning method can not only reduce the complexity of manual design, but also facilitate online deployment and adaptation to multiple scenarios. In view of the defects existing in the previous loop detection method, the present invention adopts an aggregator with better performance and improves the triple-based metric learning method. In order to prevent catastrophic forgetting of the model, the present invention proposes a biologically based long-short-term triple memory module, which not only realizes lifelong learning, but also facilitates comparative learning to improve the learning efficiency of the model, thereby realizing the cross-regional scene recognition function for the robot. Summary of the invention

[0008] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a robot scene recognition method based on lifelong learning.

[0009] In a first aspect, the present invention provides a robot scene recognition method based on lifelong learning, the method comprising:

[0010] Step S1: Obtain a scene image captured by a robot camera;

[0011] Step S2: preprocessing the scene image;

[0012] Step S3: Input the preprocessed scene image into the long short-term triple memory module; randomly sample an anchor data I from the long short-term triple memory module a As a training scene, multiple scenes with the same content as training scene I are randomly sampled. a Positive data belonging to the same scenario As a learning model, and multiple training scenarios I a Negative data belonging to different scenarios

[0013] From training scene I a , multiple positive examples Multiple negative examples Constitute a small sample set;

[0014] Step S4: Input the data in the small sample set into the scene recognition network and output the small sample set f corresponding to the feature vector a According to the cosine similarity between feature vectors, find the feature vector f of the same scene that is most dissimilar p and the most similar different scene feature vector f n , the feature vector f corresponding to the training scene a Constitute a triple after deep mining. Then according to the triple after deep mining f a , f p , f n Calculate the loss function;

[0015] The scene recognition network includes a VGG16 feature extractor and a NetVLAD aggregator;

[0016] The NetVLAD aggregator specifically achieves feature aggregation through clustering and residual summation.

[0017] The loss function of the scene recognition network is Where t represents the loss function of the tth environment, λ1, λ2 are hyperparameters, and L hard-triplet is the triplet loss function for deep mining, L RMAS is the relational memory-aware synaptic loss function, L KD is the distribution-based knowledge distillation loss function.

[0018] The triplet loss function L of the deep mining hard-triplet The details are as follows:

[0019]

[0020] in is the cosine similarity between the negative image feature and the anchor image feature output by the scene recognition network, is the similarity between the jth positive image feature output by the scene recognition network and the anchor image feature, and δ is an artificially set hyperparameter.

[0021] The relational memory-aware synapse (RMAS) loss function L RMAS The details are as follows:

[0022]

[0023]

[0024] in N represents the i-th parameter when the scene recognition network learns the t-th task, t is the number of samples in the training batch, It represents the Gram matrix composed of triple samples when training to task t, and F is the Frobenius norm.

[0025] The distribution-based knowledge distillation loss function L KD The details are as follows:

[0026]

[0027] in represents the i-th feature vector extracted by the scene recognition network in the t-th environment, and d is Dimension.

[0028] Step S5: updating the memory of the long-term and short-term triple memory modules;

[0029] The oldest scene in the short-term memory module is stored in the panoramic memory module. If the space in the panoramic memory module is full, the data inside the panoramic memory module is randomly replaced based on the ratio of the space capacity to the amount of data read in as the probability of update, to ensure that the panoramic memory module can represent the current complete environment. After entering a new environment, the panoramic memory module of the previous environment is stored in the long-term memory module in proportion to ensure that the long-term memory module can represent multiple environments so far.

[0030] Step S6: Repeat S1 to S5 to complete the training of the scene recognition network.

[0031] Step S7: Input the scene image into the scene recognition network and output the feature vector f'.

[0032] Step S8: Compare f' with the existing scene feature vectors in the database, find the scene with the highest similarity as the recognition result, and realize the scene recognition function.

[0033] In a second aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method described.

[0034] In a third aspect, the present invention provides a computing device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the described method is implemented.

[0035] The beneficial effects of the present invention are:

[0036] The present invention belongs to the field of lifelong learning and provides a robot scene recognition method, which solves the robot's ability to recognize scenes in a constantly changing environment and avoids the problem of forgetting old scenes when learning new scenes, that is, solving the problem of catastrophic forgetting. This method improves the robot's scene recognition accuracy and lifelong learning ability by integrating advanced feature extraction technology and aggregation technology, combining long-term and short-term memory mechanisms, and intelligent knowledge distillation and memory-aware synaptic strategies.

[0037] In terms of network architecture, the present invention uses VGG16 as a feature extractor and NetVLAD technology as an aggregator, which can provide better aggregation effects when processing continuously shot scene images compared to traditional pooling technology. NetVLAD forms a richer global description through cluster analysis and adds the residuals of all local features to the cluster center point to adapt to multi-angle and multi-scale video image data.

[0038] In terms of feature learning, the present invention uses a deep mining triplet loss function to optimize the model's learning ability for different scenarios by learning difficult samples.

[0039] In terms of lifelong learning, in order to prevent catastrophic forgetting when learning new scenes, the present invention designs a long-short-term triple memory module. This module contains a short-term memory queue, a panoramic memory bank, and a long-term memory bank. When passing through different environments, the memory content is dynamically adjusted to ensure the robot's memory ability. This method ensures that the model can quickly adapt to new environments while maintaining memory of old environments. At the same time, the distribution-based knowledge distillation method and memory-aware synapses are used to further enhance the stability of the model and alleviate the problem of catastrophic forgetting.

[0040] The lifelong learning robot scene recognition method of the present invention has significant performance improvements, especially in complex and changing environments. Compared with the prior art, the present method can provide higher recognition accuracy and better long-term learning ability. Through application and testing on the multi-season Nordland and multi-lighting conditions Oxford datasets, the present invention demonstrates its powerful recognition ability for different scenes, as well as its ability to balance the retention of old knowledge and the learning of new knowledge in lifelong learning scenarios. These characteristics make the present invention have broad application prospects in practical applications, such as autonomous driving, service robots and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 It is a flow chart of the method provided by an embodiment of the present invention.

[0043] Figure 2 This is a schematic diagram of the long short-term triple memory module.

[0044] Figure 3 The same scene is identified in different seasons.

[0045] Figure 4 It is similar to the same season but not the same scene. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0047] like Figure 1 As shown, the robot scene recognition method based on lifelong learning provided by the embodiment of the present invention has the following advantages:

[0048] 1. Strengthening lifelong learning

[0049] Traditional lifelong learning systems may face the problems of forgetting and model degradation when processing long-term data. This paper introduces the RMAS and KD methods. RMAS ensures that key parameters will not be completely updated by calculating the importance of parameters; KD constrains the global large-scale update of the model by comparing the model of the current task with the model of the previous task. Together, the two methods prevent catastrophic forgetting. These two methods enhance the robustness of the model and maintain its excellent performance in past tasks. At the same time, in order to improve the plasticity of the model, its learning efficiency in future tasks is guaranteed.

[0050] 2. Experience replay of long-term and short-term memory modules

[0051] The present invention also proposes a biologically based memory module, which imitates the human memory system and designs a triple memory module that combines long-term and short-term memory. This module is composed of short-term memory with timeliness, representative panoramic memory and global long-term memory. Such a structure helps the model learn long-term multi-scenario tasks. This module not only improves the memory capacity of the model, but also can effectively handle changes and noise in sequence data, thereby improving the performance of the model in lifelong learning.

[0052] 3. Optimization training of triplet loss function optimized by Hard Mining

[0053] The triplet loss function optimized by Hard Mining helps to better reduce the intra-class distance and expand the inter-class distance during training, simplify calculations, pay more attention to the whole, and achieve more robust performance when facing class imbalance or the introduction of new categories. Compared with the common triplet loss function, which can only learn from a pair of samples, the triplet loss function optimized by Hard Mining considers more holistic and comprehensive data, which can improve the robustness of the model. The use of this loss function provides a stronger optimization signal in the lifelong learning framework, allowing the model to better adapt to new tasks and reduce the impact on old tasks.

[0054] 4. Global feature aggregation of NetVLAD

[0055] The introduction of NetVLAD technology further improves the model's ability to express input data by aggregating and representing local features. The introduction of NetVLAD helps reduce redundant information, improve the robustness and discrimination of features, and thus enhance the effectiveness and interpretability of the model for input data. NetVLAD aggregates local features to retain richer detail information in the image. At the same time, the aggregation of local features of the image has a certain degree of spatial invariance, which is suitable for processing multi-scale, multi-angle video image data, and is more in line with the shooting situation of the robot.

[0056] 5. Improvement of overall performance

[0057] By comprehensively using RMAS, KD method, memory module, deep mining triple loss function and NetVLAD technology, the technical solution of the present invention achieves more outstanding performance in lifelong learning tasks. The model is more versatile and more adaptable to new tasks, while maintaining robust performance on old tasks. These improvements enable lifelong learning systems to better cope with complex and changing environments, providing a more reliable solution for long-term deployment and application. For the Nordland dataset, lifelong learning is deployed in its four seasons, with a performance improvement of more than 10% compared to the current best algorithm.

[0058] The robot scene recognition method based on lifelong learning of the present invention mainly implements specific functions through pytorch code. The specific implementation method of each key step will be introduced below. The method includes:

[0059] Step S1: Obtain a scene image captured by a robot camera;

[0060] Step S2: Preprocess the scene image.

[0061] First, all images are scaled to a size of 480*640 to facilitate subsequent data processing; second, the images are randomly rotated as reinforcement data, and the reinforcement data is used together with the original data for training to enhance the generalization ability of the model.

[0062] Step S3: Input the preprocessed scene image into the long short-term triple memory module; randomly sample an anchor data I from the long short-term triple memory module a As a training scene, multiple scenes with the same content as training scene I are randomly sampled. a Positive data belonging to the same scenario As a learning model, and multiple training scenarios I a Negative data belonging to different scenarios

[0063] From training scene I a , multiple positive examples Multiple negative examples Constitute a small sample set;

[0064] Step S4: Input the data in the small sample set into the scene recognition network and output the small sample set f corresponding to the feature vector a According to the cosine similarity between feature vectors, find the feature vector f of the same scene that is most dissimilar p and the most similar different scene feature vector f n , the feature vector f corresponding to the training scene a Constitute a triple after deep mining. Then according to the triple after deep mining f a , f p , f n Calculate the loss function;

[0065] The triplet sampling strategy of deep mining optimization is committed to improving the efficiency and performance of model training, with a special focus on selecting challenging sample pairs from the long-short-term triple memory module. In each training step, the strategy first randomly selects an anchor data point, and then selects the same class sample that is most dissimilar to the anchor point as the positive example, and two different class samples that are most similar to the anchor point as the negative example. The triplet composed in this way emphasizes the model's learning in distinguishing difficult samples, that is, focusing on processing heterogeneous samples with high similarity to the anchor point and samples with large intra-class differences.

[0066] Through this triple sampling method optimized for mining hard-to-distinguish examples, the model is forced to focus on learning those pairs of positive and negative samples that are difficult to distinguish, effectively promoting a clear division of the boundaries between classes. This strategy not only helps to reduce the distance between samples of the same type and enhance the compactness within a class, but also effectively expands the interval between samples of different classes, thereby significantly improving the model's ability to generalize to new scenarios. The core advantage of this method is that it accelerates the model's learning process for complex data distributions by strategically selecting training samples, which is especially suitable for application scenarios where the boundaries between sample categories are not obvious.

[0067] The long-short-term triple memory module is mainly composed of a short-term memory module, a panoramic memory module and a long-term memory module. The short-term memory module is implemented through a first-in-first-out image queue. This module imitates human short-term memory, reads data into the system one by one, and forgets it one by one over time. When the short-term memory module is full, the oldest image will be stored in the panoramic memory module. As the input of images continues to increase, the storage space of the panoramic memory module will also be filled. So we use the ratio of the storage space of the panoramic memory module to the amount of data read in as the update probability. After receiving the image input from the short-term memory module, if it is greater than the update probability, the panoramic memory module will not be updated, and if it is less than the update probability, it will be stored in a random position of the panoramic memory module. When the robot enters a new environment for training, the panoramic memory module of the past environment is stored in the long-term memory module according to the following strategy:

[0068]

[0069] in Represented as the long-term memory module at the t-th task, It represents the panoramic memory module at the t-th task, and λ is a hyperparameter that is considered.

[0070] The scene recognition network includes a VGG16 feature extractor and a NetVLAD aggregator; the image is converted into 1024-dimensional feature data through the network. After the scene recognition network training is completed, the feature data is stored in the database. The same operation is performed every time an image is input, and the cosine similarity is compared with the feature data already in the database to determine whether it is the same location, thereby realizing the function of scene recognition.

[0071] The NetVLAD aggregator specifically achieves feature aggregation through clustering and residual summation.

[0072] The feature extractor of this method uses the VGG16 network, but unlike other methods that use the generalized average pooling method (GEM) as an aggregator, this method uses the local aggregation descriptor vector method (NetVLAD) as an aggregator. GEM is essentially a generalized form of the maximum pooling layer and the average pooling layer. The specific calculation formula is as follows:

[0073]

[0074] Where X k is the input data, p k is a hyperparameter. Unlike traditional evaluation pooling or maximum pooling, GEM allows the model to learn nonlinear pooling of features. The advantage of GEM is that it can learn different feature aggregation methods by adjusting the parameter p to better adapt to the feature distribution of the data.

[0075] NetVLAD first extracts multiple local features, and then locally aggregates the local features of each frame to form a global description. Local aggregation first performs cluster analysis on the features, then assigns all local features to the cluster center, calculates the residuals of all local features and adds them up to form a global feature. The specific aggregation method is as follows:

[0076]

[0077] where x i (j) is the i-th descriptor of the j-th dimension, c k (j) is the kth cluster center point of the jth dimension, a k (x i ) indicates whether the i-th descriptor belongs to the k-th cluster.

[0078] In NetVLAD, local feature extraction and clustering are both achieved through neural networks to aggregate local features into global descriptions end-to-end.

[0079] As a pooling method, GEM has the advantage of being more flexible and can learn more complex feature aggregation methods. However, the pooling method may lose some detailed information, especially when processing image information. GEM may not be able to capture some local structures, resulting in information loss, while NetVLAD aggregates local features and retains richer details in the image. At the same time, NetVLAD aggregates local features of the image, has a certain spatial invariance, and is suitable for processing multi-scale and multi-angle video image data. Therefore, in the robot scene recognition problem, the input image is captured from the video shot by the robot, which is more suitable for using NetVLAD. At the same time, comparative experiments have also proved that NetVLAD performs better than GEM in this task.

[0080] In order to facilitate training and network structure fusion, this method uses the aggregation structure in NetVLAD instead of the traditional VLAD structure. First, the cluster center is calculated through a 1*1 convolution kernel, and softmax is used to convert it into a probability distribution to better represent the weight of local features. Then, the activation value is aggregated with the input data through matrix multiplication, and the weight of the corresponding cluster center is subtracted to emphasize the difference between the local feature and the cluster center. Finally, the VLAD representation is normalized to improve the robustness of the feature.

[0081] In terms of the loss function of network training, the triplet loss function is the most common form in similar image retrieval tasks. This method uniformly samples a triplet from the storage space, including an anchor data, a positive data of the same type as the anchor data, and a negative data of a different type from the anchor data. By calculating the cosine similarity of the triplet in the Euclidean space, the model is forced to learn the implicit features in heterogeneous data. The triplet loss function was first proposed by the Google research team for face recognition tasks. It can distinguish between similar samples of different types and can model details well. Therefore, it is also suitable for scene recognition tasks. The calculation formula is as follows:

[0082] L triplet =max(S an -S ap +δ,0)

[0083] Where S an represents the similarity between anchor data and negative data, S ap It represents the similarity between the anchor data and the positive data, and δ is a hyperparameter set artificially.

[0084] Although the triplet loss function is very suitable for this task, it also has some disadvantages: the selection of triples may not result in uniform data distribution, the model is trained in units of triples, the content of model learning is limited, the training process is unstable, and the generalization ability is not strong enough. Therefore, this method introduces hard mining triplet, taking the most dissimilar positive examples and the most similar negative examples for training to improve learning efficiency. The calculation formula is as follows:

[0085]

[0086] in is the cosine similarity between the negative image feature and the anchor image feature output by the scene recognition network, is the similarity between the jth positive image feature output by the scene recognition network and the anchor image feature, and δ is an artificially set hyperparameter.

[0087] Compared with the traditional triplet loss, the triplet loss function of deep mining pays more attention to selecting positive and negative sample pairs that are difficult to distinguish during training. This method directly targets the difficult problem by selecting the negative examples that are most similar to the anchor point and the positive examples that are least similar to the anchor point for training, thereby effectively reducing the intra-class distance and increasing the inter-class distance. This strategy not only accelerates the convergence speed of the model, but also significantly improves the generalization ability of the model on unseen data, which is especially suitable for scenarios that require fine recognition capabilities.

[0088] Next, we will introduce some designs for alleviating catastrophic forgetting. In terms of regularization, this method uses relational memory-aware synapses (RMAS) and knowledge distillation (KD). In the regularized lifelong learning method, it is believed that the model's learning process for new tasks causes changes in parameters that are very important for old tasks, which leads to catastrophic forgetting. Therefore, the purpose of MAS is to assign an importance weight to each parameter and punish its changes through regularization loss during training. RMAS is used in this method, and its calculation formula is as follows:

[0089]

[0090]

[0091] in represents the i-th parameter when learning the t-th task, sim(*), *) represents the similarity calculation, f(*) represents the scene recognition network, and I, r represents the input image data.

[0092] However, in the environment of lifelong learning, directly calculating the coefficient of RMAS requires all sample pairs of the current environment, which is not in line with the actual situation and will increase the amount of calculation. Therefore, this method will approximate it through the following formula:

[0093]

[0094] in It represents the Gram matrix composed of triple samples of the kth training step when training to task t, and F is the Frobenius norm.

[0095] The knowledge distillation of this method is different from the knowledge distillation commonly used in other methods. In the past, the knowledge distillation method for lifelong learning often used the Euclidean distance output by the previous and next models as a penalty term to ensure that the models at different times can maintain stability. However, this method is too simple, and it is calculated based on the Euclidean distance, lacking analysis of global information. Therefore, the present invention proposes a distribution-based knowledge distillation method. By calculating the dot product of the output vectors of the previous and next models to form a knowledge matrix, we use KL divergence to compare the distribution of the knowledge matrix at two times, and use this result as a penalty term to constrain the consistency of the global distribution of the previous and next models. The specific formula is as follows:

[0096]

[0097] in represents the i-th feature vector extracted in the t-th environment, and d is This method of using KL divergence not only optimizes the knowledge transfer process, enabling the new model to learn knowledge from the old model more effectively, but also improves the stability and generalization ability of the model in a multi-task learning environment. In addition, by adjusting the proportion of KL divergence loss in the total loss function, the relationship between new knowledge learning and old knowledge retention can be flexibly balanced, further improving the scene recognition performance of lifelong learning robots in complex environments.

[0098] In order to achieve lifelong learning while avoiding increasing the storage burden, the model usually adopts the memory replay method, which saves images of some scenes in the past for training to achieve long-term learning. In this invention, we designed a long-term and short-term triple memory module based on biology and imitating the human memory system to achieve the training process of lifelong learning. At the beginning, the video information stored by the robot is divided into images and stored in a first-in-first-out queue, which is regarded as the robot's short-term memory module;

[0099] When the short-term memory module is full, the oldest image will be stored in the panoramic memory module. As the number of images input increases, the storage space of the panoramic memory module will also be filled. Therefore, we use the ratio of the storage space of the panoramic memory module to the amount of data read in as the update probability. After receiving the image input from the short-term memory module, if it is greater than the update probability, the memory module will not be updated, and if it is less than the update probability, it will be stored in a random position of the panoramic memory module. When the robot enters a new environment for training, the panoramic memory module of the past environment is stored in the long-term memory module according to the following strategy:

[0100]

[0101] in Represented as the long-term memory module at the t-th task, It represents the panoramic memory module at the t-th task, and λ is a hyperparameter that is considered.

[0102] Through the triple memory module, we successfully stored three types of time-sensitive memories in different spaces. When training, we selected images from the entire memory module to ensure image diversity and improve learning efficiency. In addition, we only use the long-term memory module for knowledge distillation to ensure that knowledge distillation is only performed through knowledge learned by the previous and next models.

[0103] Finally, combining the above modules, the comprehensive loss function is as follows:

[0104]

[0105] When comprehensively considering knowledge distillation and memory-aware mechanisms, we introduce a strategy that combines KL divergence (Kullback-Leibler divergence) with knowledge distillation (KD), aiming to optimize the model training process in the lifelong learning framework to alleviate the catastrophic forgetting phenomenon that occurs when the model faces continuous learning tasks.

[0106] The relational memory-aware synapse (RMAS) strategy constructs a 3*3 similarity matrix based on the triplets sampled in each training step, which reflects the similarity between the anchor data, positive data, and two negative data after network processing. By calculating the Frobenius norm of this matrix and taking partial derivatives of the parameters in the model, we can approximate the importance of the model parameters in the current training process, and use this information to protect key parameters in future learning to avoid the loss of important knowledge.

[0107] In addition, in order to effectively reduce forgetting during lifelong learning, we use distribution-based knowledge distillation technology. This method pays special attention to the knowledge transfer between two consecutive training steps of the model. By calculating the KL divergence of the probability distribution of the model output in the two stages, it accurately measures and minimizes information loss from a global perspective, thus ensuring the ability of lifelong learning.

[0108] Step S5: using the triple memory module to update the long and short-term memory module;

[0109] like Figure 2 The oldest scene in the long-term and short-term memory modules is stored in the panoramic memory module. If the space in the panoramic memory module is full, the data inside the panoramic memory module is randomly replaced based on the ratio of the space capacity to the amount of data read in as the probability of update to ensure that the panoramic memory module can represent the current complete environment. After entering a new environment, the panoramic memory module of the previous environment is stored in the long-term memory module in proportion to ensure that the long-term memory module can represent multiple environments so far.

[0110] Step S6: Repeat S1 to S5 to complete the training of the scene recognition network.

[0111] Step S7: Input the scene image into the scene recognition network and output the feature vector f'.

[0112] Step S8: Compare f' with the existing scene feature vectors in the database, find the scene with the highest similarity as the recognition result, and realize the scene recognition function.

[0113] Experimental Results

[0114] For scene recognition, we use the recall rate at 100% precision as the basic evaluation indicator; for multi-scene lifelong learning, we construct a t*t evaluation matrix P for the recognition task with t environments. t*t , where P i，j It represents the system's ability to recognize scenes in the jth environment after learning the i-th environment. The evaluation indicator is the recall rate under the 100% precision mentioned above. We derive the final average performance indicator based on the evaluation matrix

[0115]

[0116] We evaluate the experimental results on the Nordland dataset with multiple seasonal environments and the Oxford Robotcar dataset with multiple lighting conditions.

[0117] Table 1 Final results

[0118] Nordland Oxford NetVLAD 0.615 0.411 AirLoop 0.631 0.461 Our Approach 0.715 0.540

[0119] Figure 3-4 These are images recorded during the experiment.

[0120] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A robot scene recognition method based on lifelong learning, characterized in that: The method comprises: Step S1: Obtain a scene image captured by a robot camera; Step S2: preprocessing the scene image; Step S3: Input the preprocessed scene image into the long short-term triple memory module, and randomly sample an anchor data I from the long short-term triple memory module. a As a training scene, multiple scenes with the same content as training scene I are randomly sampled. a Positive data belonging to the same scenario As a learning model, and multiple training scenarios I a Negative data belonging to different scenarios From training scene I a , multiple positive examples Multiple negative examples Constitute a small sample set; The long-term and short-term triple memory module includes a short-term memory module, a panoramic memory module and a long-term memory module; the short-term memory module is implemented by a first-in-first-out image queue; the panoramic memory module is formed by outputting images from the short-term memory module and performing internal screening; the long-term memory module is obtained by filtering the panoramic memory module of the historical scene; Step S4: Input the data in the small sample set into the scene recognition network and output the small sample set f corresponding to the feature vector a , According to the cosine similarity between feature vectors, find the feature vector f of the same scene that is most dissimilar p and the most similar different scene feature vector f n , the feature vector f corresponding to the training scene a Form a triple after deep mining; Step S5: Update the memory of the long-term and short-term triple memory module; specifically: The oldest scene in the short-term memory module is stored in the panoramic memory module. If the space of the panoramic memory module is full, the data inside the panoramic memory module is randomly replaced according to the ratio of the space capacity to the amount of data read in as the probability of update, so as to ensure that the panoramic memory module can represent the current complete environment; after entering the new environment, the panoramic memory module of the previous environment is stored in the long-term memory module according to the ratio, so as to ensure that the long-term memory module can represent multiple environments so far; Step S6: Repeat steps S1 to S5 to complete the training of the scene recognition network; Step S7: input the scene image into the scene recognition network and output the feature vector f'; Step S8: Compare f' with the existing scene feature vectors in the database, find the scene with the highest similarity as the recognition result, and realize the scene recognition function.

2. The method according to claim 1, characterized in that The scene recognition network includes a VGG feature extractor and a VLAD aggregator.

3. The method according to claim 1, characterized in that The VGG feature extractor adopts the VGG16 feature extractor; the VLAD aggregator adopts the NetVLAD aggregator; the NetVLAD aggregator specifically realizes feature aggregation through clustering and residual summation.

4. The method according to claim 1, characterized in that The loss function of the scene recognition network is as follows: Where t represents the loss function of the tth environment, λ1, λ2 are hyperparameters, and L hard-triplet is the triplet loss function for deep mining, L RMAS is the relational memory-aware synaptic loss function, L KD is the distribution-based knowledge distillation loss function; The triplet loss function L of the deep mining hard-triplet The details are as follows: in is the cosine similarity between the negative image feature i and the anchor image feature output by the scene recognition network, is the similarity between the jth positive image feature and the anchor image feature output by the scene recognition network, and δ is a manually set hyperparameter; The relational memory-aware synaptic loss function L RMAS The details are as follows: in N represents the i-th parameter when the scene recognition network learns the t-th task, t is the number of samples in the training batch, represents the Gram matrix composed of triple samples when training to task t, and F is the Frobenius norm; The distribution-based knowledge distillation loss function L KD The details are as follows: in represents the i-th feature vector extracted by the scene recognition network in the t-th environment, and d is Dimension.

5. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 4.

6. A computing device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 4 is implemented.

Images