Scene recognition method for robots based on lifelong learning

Abstract

To provide a scene recognition method for robots based on lifelong learning.SOLUTION: By integrating advanced feature extraction and aggregation technologies, the present invention combines mechanisms of long-term memory and short-term memory with intelligent knowledge distillation and a memory-perception synapse policy, thereby improving the recognition accuracy and lifelong learning ability of a robot for scenes. The lifelong-learning-robot scene recognition method according to the present invention exhibits significantly improved performance. In particular, in complex and changing environments, this method can provide higher recognition accuracy and superior long-term learning ability compared with conventional techniques. Application and testing on the Nordland dataset across multiple seasons and the Oxford dataset under multiple illumination conditions have demonstrated that the invention achieves strong discriminative capability for different scenes and successfully harmonizes the retention of old knowledge with the acquisition of new knowledge in lifelong learning scenes.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention belongs to the field of robotics technology, and more particularly to a scene recognition method for a robot based on lifelong learning. [Background technology]

[0002] Broadly speaking, the loop closure detection task corresponds to the scene recognition task, and the goal is to locate the current scene. Loop closure detection is an important component of modern synchronous localization and mapping (SLAM) systems, and is mainly used to eliminate the accumulated offset error after loop closure detection, improving the accuracy of mapping and realizing the scene recognition function of the robot. The goal of loop closure detection is to recognize an environment when given an image of it if it has been experienced before, thereby giving the robot scene recognition capabilities.

[0003] Traditional loop closure detection methods use manual feature calculation, such as using algorithms like HoG and SIFT to generate visual language and then construct a bag of words (BoW) model based on it. With the development of artificial intelligence, loop closure detection has since been realized using deep learning techniques. The main model consists of two parts: a feature extractor that extracts several features from the image and an aggregator that integrates multiple features. Finally, the results output by the aggregator are compared with each other, converting the loop closure detection problem into an image search problem.

[0004] Although this method can theoretically realize robot scene recognition, it faces many other problems in practical applications. Since the robots deployed in actual projects do not operate in a single scene, the robot's scene recognition ability needs to adapt to an increasing number of environments depending on various situations. However, the robot's memory capacity is limited, and without special design, it will only lead to the following two results. 1. After learning a new scene, the robot forgets its ability to recognize the old scene, which is catastrophic forgetting. 2. The robot stores all scene data, but as the robot is used, the storage overhead becomes huge, making online deployment difficult. Therefore, to solve the above problems, the present invention proposes a scene recognition method for robots based on lifelong learning.

[0005] Lifelong learning for robots is a learning process that mimics that of humans. Humans are not isolated in the learning process, but constantly accumulate knowledge learned in the past and use it to learn more knowledge. They forget some knowledge, but as time goes by, they learn more and more knowledge. Lifelong learning has three characteristics. 1. Continuous learning. 2. Knowledge is stored in a knowledge base. 3. It helps us use past knowledge to learn future knowledge. Applying lifelong learning to robot scene recognition allows the robot to learn more scenes without catastrophic forgetting.

[0006] In recent years, several methods based on lifelong learning have been used to realize robot scene recognition, but they suffer from the following shortcomings. First, the simple pooling method used in the aggregator cannot adapt well to continuously captured scene images. Second, comparative learning typically uses a three-group loss function, which makes the learning process unstable and inefficient. Second, the design of knowledge distillation is too simple and does not take into account the overall distillation effect of the model. Finally, the design of the memory module in memory recall is not detailed enough, resulting in poor performance. Summary of the Invention [Problem to be solved by the invention]

[0007] This invention designs a loop closure detection method for robot scene recognition based on lifelong learning. This method not only reduces the complexity of manual design but also facilitates online deployment and adaptation to multiple scenes. To address the shortcomings of previous loop closure detection methods, this invention employs a better-performing aggregator and improves the three-group metric learning method. To prevent catastrophic forgetting in the model, this invention proposes a biologically based long-term and short-term triple memory module that realizes lifelong learning while promoting comparative learning to improve the model's learning efficiency, thereby realizing scene recognition capabilities beyond the realm of robots.

[0008] SUMMARY OF THE INVENTION An object of the present invention is to overcome the shortcomings of the prior art and to provide a scene recognition method for a robot based on lifelong learning. [Means for solving the problem]

[0009] In a first aspect, the present invention provides a scene recognition method for a robot based on lifelong learning, the method including: Step S1: A scene image captured by a robot camera is acquired. Step S2: Preprocessing is performed on the scene image. Step S3: The preprocessed scene image is input to the long-term and short-term triple memory module. The long-term and short-term triple memory module jointly and randomly selects anchor data I a is sampled as a training scene, and then training scene I a and multiple positive sample data belonging to the same scene I pi Randomly sampled and used as training samples, and training scene I a and multiple negative sample data belonging to different scenes I ni are sampled randomly. Learning Scene I a , multiple positive sample data I pi , multiple negative sample data I ni A small sample set is constructed from Step S4: Input the data in the small sample set into the scene recognition network, and generate a small sample set f of corresponding feature vectors. a ,f pi ,f ni The cosine similarity between the feature vectors is used to determine the least similar feature vector f p The feature vector f of a different scene that is most similar to n and find the feature vector f a and one deep mined group of three. Three deep mined groups f a ,f p ,f n The loss function is calculated by The scene recognition network includes a VGG16 feature extractor and a NetVLAD aggregator. The NetVLAD aggregator specifically realizes feature aggregation through clustering and residual summation.

[0010] The loss function of the scene recognition network is:

[0011]

number

[0012] where t denotes the loss function for the t-th environment, λ1 and λ2 are hyperparameters, and L hard-triplet are the loss functions of the three groups of deep mining, and L RMAS is the relational memory-perception synaptic loss function, and L KD is the distribution-based knowledge distillation loss function.

[0013] The loss function L of the three groups of deep mining hard-triplet Specifically, the following are the details:

[0014]

number

[0015] Here, symbol (2-1) is the cosine similarity between the i-th negative sample image feature output by the scene recognition network and the anchor image feature, symbol (2-2) is the similarity between the j-th positive sample image feature output by the scene recognition network and the anchor image feature, and δ is an artificially set hyperparameter.

[0016] The relational memory-perceptual synapse (RMAS) loss function L RMAS Specifically, the following are the details:

[0017]

number

[0018] Here, symbol (3-1) denotes the i-th parameter when the scene recognition network learns the t-th task, and N t denotes the number of samples in the training batch, the symbol (4-1) denotes the Gram matrix consisting of three groups of samples when training up to task t, and F is the Frobenius norm.

[0019] The knowledge distillation loss function L based on the distribution KD Specifically, the following are the details:

[0020]

number

[0021] Here, in equation (6), symbol (6-1) represents the i-th feature vector extracted by the scene recognition network in the t-th environment, and d is the dimension of symbol (6-1).

[0022] Step S5: The long-term and short-term triple memory module is updated. The oldest scene in the short-term memory module is written to the panoramic memory module, and when the space of the panoramic memory module is full, the data in the panoramic memory module is randomly swapped with the ratio of the space capacity to the amount of data loaded as the update probability, so that the panoramic memory module can represent the complete current environment. After entering a new environment, the panoramic memory module of the previous environment is written proportionally to the long-term memory module, so that the long-term memory module can represent multiple environments up to the present.

[0023] Step S6: Steps S1 to S5 are repeated to complete the learning of the scene recognition network.

[0024] Step S7: The scene image is input to the scene recognition network, and a feature vector f' is output.

[0025] Step S8: Compare f' with the existing scene feature vectors in the database, and find the scene with the highest similarity as the recognition result, thereby realizing the scene recognition function.

[0026] In a second aspect, the present invention provides a computer readable storage medium having stored thereon a computer program which, when executed in a computer, causes the computer to carry out the method.

[0027] In a third aspect, the present invention provides a computer apparatus, comprising a memory and a processor, storing executable code in said memory, said computer apparatus implementing the method when said processor executes said executable code.

[0028] The beneficial effects of the present invention are as follows: This invention provides a robot scene recognition method based on lifelong learning, which solves the problem of a robot's ability to recognize scenes in a changing environment, while avoiding the problem of forgetting old scenes when learning new ones, i.e., solving the problem of catastrophic forgetting. By integrating advanced feature extraction and aggregation techniques, this method combines long-term memory mechanisms, short-term memory mechanisms, and intelligent knowledge distillation and memory-sensing synapse strategies to improve the robot's scene recognition accuracy and lifelong learning ability.

[0029] In terms of network architecture, this invention uses VGG16 as the feature extractor and NetVLAD technology as the aggregator, which can provide better aggregation results when processing continuously captured scene images compared to traditional pooling techniques. NetVLAD performs clustering analysis and sums the residuals of all local features relative to cluster centers to form a richer global description, adapting to multi-angle and multi-scale video image data.

[0030] In terms of feature learning, the present invention uses three groups of loss functions from deep mining to learn difficult samples and optimize the model's learning ability for different scenes.

[0031] In terms of lifelong learning, to prevent catastrophic forgetting when learning new scenes, the present invention designs a long-term, triple-layer memory module. This module includes a short-term memory queue, a panoramic memory bank, and a long-term memory bank, and dynamically adjusts memory content when experiencing different environments to ensure the robot's memory capacity. This method allows the model to not only quickly adapt to new environments, but also retain memories of old environments. At the same time, the distribution-based knowledge distillation method and memory-sensing synapses are used to further enhance the stability of the model and mitigate the problem of catastrophic forgetting.

[0032] The robot scene recognition method based on lifelong learning of the present invention has significantly improved performance, providing higher recognition accuracy and better long-term learning capabilities compared to conventional techniques, especially in complex and varied environments. Through application and testing on the Nordland dataset across multiple seasons and Oxford datasets under multiple lighting conditions, the present invention has demonstrated strong discrimination capabilities for different scenes and successfully balanced the retention of old knowledge with the acquisition of new knowledge in lifelong learning scenarios. These features make the present invention promising for practical applications in fields such as autonomous driving and service robots. [Brief explanation of the drawings]

[0033] In order to more clearly describe the technical solutions of the present invention, the following briefly describes the drawings that need to be used in the embodiments. The drawings in the following description are only some embodiments of the present invention, and it is obvious that those skilled in the art can obtain other drawings based on these drawings without any creative efforts. [Figure 1] 1 is a flowchart of a method provided by an embodiment of the present invention. [Figure 2] FIG. 1 is a schematic diagram of a long-term and short-term triple storage module. [Figure 3] The same scene recognized in different seasons. [Figure 4] Similar but different scenes from the same season. DETAILED DESCRIPTION OF THE INVENTION

[0034] Below, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the drawings in the embodiments of the present invention.

[0035] As shown in FIG. 1, the robot scene recognition method based on lifelong learning provided by the embodiment of the present invention has the following advantages: 1. Strengthening lifelong learning Traditional lifelong learning systems can face problems of forgetting and model degeneration when processing long-term data. This invention introduces the RMAS and KD methods. RMAS calculates parameter importance to prevent important parameters from being completely updated. KD constrains the global update of the model by comparing the model for the current task with the model for the previous task, and the two methods jointly prevent catastrophic forgetting. These two methods enhance the stability of the model and maintain its excellent performance in past tasks. At the same time, they improve the plasticity of the model, ensuring learning efficiency in future tasks. 2. Long-term memory module experience recall The present invention also proposes a biologically based memory module, mimicking the human memory system and designing a triple memory module that combines long-term and short-term memory. The module is composed of a short-term memory with time-dependent properties, a panoramic memory with representational properties, and a long-term memory with global properties. This structure helps the model learn for long-term, multi-scene tasks. This module not only improves the model's memory ability, but also effectively handles changes and noise in sequence data, improving the model's performance in lifelong learning. 3. Optimization training of three groups of loss functions optimized by hard mining The hard-mining optimized three-group loss function better reduces intra-class distance and increases inter-class distance during training, simplifies calculations, focuses more on the whole, and achieves more stable performance when faced with class imbalance or the introduction of new classes. Unlike common three-group loss functions that only learn from paired samples, the hard-mining optimized three-group loss function considers more holistic and comprehensive data to improve model stability. The use of this loss function provides a stronger optimization signal in a lifelong learning framework, making the model better suited to new tasks and less susceptible to old tasks. 4. Global feature aggregation in NetVLAD The NetVLAD technology is introduced to further improve the model's representation capability for input data through local feature aggregation and representation. The introduction of NetVLAD reduces redundant information, improves feature stability and discriminability, and improves the model's effectiveness and interpretability for input data. NetVLAD aggregates local features to retain richer details in images. At the same time, it aggregates local image features and has a certain degree of spatial invariance, making it suitable for processing multi-scale, multi-angle video image data and more suitable for robot shooting situations. 5. Improved overall performance By integrating RMAS, KD algorithms, memory modules, and deep mining loss functions, as well as NetVLAD technology, the technical solution of the present invention achieves superior performance in lifelong learning tasks. The model is more versatile and more adaptable to new tasks, while maintaining stable performance in old tasks. These improvements enable the lifelong learning system to better adapt to complex and changing environments, providing a more reliable solution for long-term deployment and application. Lifelong learning was deployed on the Nordland dataset across four seasonal environments, achieving a performance improvement of more than 10% compared to current best algorithms.

[0036] The robot scene recognition method based on lifelong learning of the present invention mainly realizes specific functions through Pytorch code, and then introduces the specific implementation methods of each key step, including: Step S1: A scene image captured by a robot camera is acquired. Step S2: Preprocessing is performed on the scene image. First, all images are scaled to 480*640 size to facilitate subsequent data processing. Second, the images are randomly rotated to create enhanced data, which are then used together with the original data for training to improve the generalization ability of the model. Step S3: The preprocessed scene image is input to the long-term and short-term triple memory module. The long-term and short-term triple memory module jointly and randomly selects anchor data I ais sampled as a training scene, and then training scene I a and multiple positive sample data belonging to the same scene I pi (I pi of" pi "of" i "teeth" p ” subscript) are randomly sampled as training samples, and training scene I a and multiple negative sample data belonging to different scenes I ni (I ni of" ni "of" i "teeth" n " subscript). Learning Scene I a , multiple positive sample data I pi , multiple negative sample data I ni A small sample set is constructed from Step S4: Input the data in the small sample set into the scene recognition network, and generate a small sample set f of corresponding feature vectors. a ,f pi ,f ni Output (f pi of" pi "of" i "teeth" p " is a subscript, f ni of" ni "of" i "teeth" n ” subscript), the cosine similarity between feature vectors is used to find the least similar feature vector f p The feature vector f of a different scene that is most similar to n and find the feature vector f a and one deep mined group of three. Three deep mined groups f a ,f p ,f n The loss function is calculated by

[0037] The three groups of sampling strategies for deep mining optimization focus on improving model learning efficiency and performance, particularly focusing on selecting challenging sample pairs from long-term and short-term triple memory modules. At each learning step, the strategy first randomly selects an anchor data point, then selects the similar sample that is least similar to the anchor point as the positive sample, and selects the two dissimilar samples that are most similar to the anchor point as the negative samples. These three groups emphasize model learning in distinguishing difficult samples, i.e., focusing on heterogeneous samples that are highly similar to the anchor point and samples with large intra-class differences.

[0038] Through these three groups of sampling methods for optimizing hard-to-separate sample mining, the model can focus on learning those hard-to-separate positive and negative sample pairs, effectively promoting clearer division of class boundaries. This strategy not only reduces the distance between similar samples and strengthens intra-class closeness, but also effectively widens the spacing between different types of samples, thereby significantly improving the model's ability to generalize to new scenes. The core advantage of this method is that by strategically selecting training samples, it accelerates the model's learning process for complex data distributions, and is particularly suitable for application scenes where the sample class boundaries are not clear.

[0039] The long- and short-term triple memory module is mainly composed of a short-term memory module, a panoramic memory module, and a long-term memory module. The short-term memory module is implemented using a first-in, first-out image sequence. This module mimics human short-term memory, sequentially loading data into the system and gradually forgetting it over time. When the short-term memory module becomes full, the oldest image is written to the panoramic memory module. As the number of images increases, the memory area of ​​the panoramic memory module becomes full. Therefore, the ratio of the memory area of ​​the panoramic memory module to the amount of data loaded is defined as the update probability. When an image is received from the short-term memory module, if the random number corresponding to the image is greater than this update probability, the panoramic memory module is not updated; if it is less than the update probability, the image is written to a random location in the panoramic memory module. When the robot enters a completely new environment and learns, the panoramic memory module of the previous environment is written to the long-term memory module according to the following policy:

[0040]

number

[0041] Here, symbol (7-1) denotes the long-term memory module for the t-th task, symbol (7-2) denotes the panoramic memory module for the t-th task, and λ is a hyperparameter to be set.

[0042] The scene recognition network includes a VGG16 feature extractor and a NetVLAD aggregator. Images are processed through the network to generate 1024-dimensional feature data. After the scene recognition network has completed its training, the feature data is stored in a database. Each time an image is input, the network performs the same operation, comparing the cosine similarity with the existing feature data in the database to determine whether the image is from the same location, thereby achieving scene recognition.

[0043] The NetVLAD aggregator specifically achieves feature aggregation through clustering and residual summation. The feature extractor in our method uses a VGG16 network, but while other methods use the generalized average pooling method (GEM) as the aggregator, our method uses the locally aggregated descriptor vector method (NetVLAD) as the aggregator. GEM is essentially a generalized form of the max pooling layer and the average pooling layer, and the specific calculation formula is as follows:

[0044]

number

[0045] where X k is the input data, p k is a hyperparameter. Unlike traditional reputation pooling or max pooling, GEM allows the model to learn a nonlinear feature pooling method. The advantage of using GEM is that by adjusting the parameter p, we can learn a different feature aggregation method that better suits the feature distribution of the data.

[0046] NetVLAD first extracts multiple local features, then aggregates the local features of each frame locally to form a global description. Local aggregation involves first performing clustering analysis on the features, then assigning all local features to cluster centers, and then calculating and summing the residuals of all local features to form a global feature. The specific aggregation method is as follows:

[0047]

number

[0048] where x i (j) is the ith descriptor of the jth dimension, and c k (j) is the kth cluster center point in the jth dimension, and a k (x i ) indicates whether the i-th descriptor belongs to the k-th cluster.

[0049] In NetVLAD, both local feature extraction and clustering use neural networks to aggregate local features into a global description end-to-end.

[0050] As a type of pooling method, GEM has the advantage of being more flexible and able to learn more complex feature aggregation methods, but pooling methods can lose some detailed information. Especially when processing image information, GEM may not be able to capture some local structures, resulting in information loss. Meanwhile, NetVLAD aggregates local features and retains richer details in the image. At the same time, NetVLAD aggregates local image features and has a certain degree of spatial invariance, making it suitable for processing multi-scale, multi-angle video image data. Therefore, in robot scene recognition problems, where the input images are clipped from videos captured by the robot, NetVLAD is more suitable. At the same time, comparative tests have demonstrated that NetVLAD outperforms GEM in this task.

[0051] To facilitate the integration of training and network structure, our method adopts the aggregation structure in NetVLAD instead of the traditional VLAD structure. First, we calculate cluster centers using a 1*1 convolution kernel, and then convert them into probability distributions using softmax to display the weights of local features in more detail. Then, we aggregate the activation values ​​and input data using matrix multiplication and subtract the corresponding cluster center weights to emphasize the differences between local features and cluster centers. Finally, we perform a normalization process on the VLAD display to improve feature stability.

[0052] The three-group loss function is the most common form in network learning loss functions and similar image search tasks. This method uniformly samples three groups from the memory space, including one anchor data, one positive sample data similar to the anchor data, and one negative sample data dissimilar to the anchor data. By calculating the cosine similarity of the three groups in Euclidean space, the model learns hidden features in dissimilar data. The three-group loss function was first proposed by Google researchers for face recognition tasks, and is capable of distinguishing dissimilar similar samples and accurately modeling details, so it can also be applied to scene recognition tasks. The calculation formula is as follows:

[0053]

number

[0054] S an indicates the similarity between the anchor data and the negative sample data, and S ap indicates the similarity between the anchor data and the positive sample data, and δ is an artificially set hyperparameter.

[0055] While the three-group loss function is well suited to this task, it does have some drawbacks. The selection of three groups leads to uneven data distribution, and training on a three-group basis limits the model's learning content, results in unstable representation of the learning process, and insufficient generalization ability. Therefore, this method introduces a deep mining triplet (hard mining triplet) to improve learning efficiency by training using the least similar positive samples and the most similar negative samples. The calculation formula is as follows:

[0056]

number

[0057] Here, symbol (11-1) is the cosine similarity between the i-th negative sample image feature output by the scene recognition network and the anchor image feature, symbol (11-2) is the similarity between the j-th positive sample image feature output by the scene recognition network and the anchor image feature, and δ is an artificially set hyperparameter.

[0058] Compared with the traditional three-group loss functions, the three-group loss functions for deep mining focus on selecting pairs of positive and negative samples that are difficult to distinguish during the training process. This method directly addresses this challenge by selecting the negative samples that are most similar to the anchor points and the positive samples that are least similar, thereby effectively reducing the intra-class distance and increasing the inter-class distance. This method not only accelerates the convergence speed of the model, but also significantly improves the model's generalization ability to unseen data, making it particularly applicable to scenes that require fine-grained discrimination.

[0059] Next, we explain several designs to mitigate catastrophic forgetting. For regularization, our method uses the relational memory-perceptual synapse (RMAS) and knowledge distillation (KD) techniques. In regularization-based lifelong learning methods, parameters important to old tasks change as the model learns new tasks, which is thought to cause catastrophic forgetting. The purpose of RMAS is to assign importance weights to each parameter and penalize those changes through regularization loss during the learning process. Our method uses RMAS, whose calculation formula is as follows:

[0060]

number

[0061] Here, the symbol (12-1) indicates the i-th parameter when learning up to the t-th task, and sim( * , * ) indicates the similarity calculation, and (f( * )) denotes the scene recognition network, and I and I' denote the input image data. However, directly calculating the RMAS coefficients in a lifelong learning environment requires all sample pairs in the current environment, which is unrealistic and increases the amount of calculation required. Therefore, this method is approximated by the following formula:

[0062]

number

[0063] Here, symbol (14-1) denotes the Gram matrix consisting of three groups of samples at the kth learning step when learning up to task t, and F is the Frobenius norm.

[0064] The knowledge distillation method used in this study differs from other commonly used methods. Traditional knowledge distillation techniques in lifelong learning use the Euclidean distance output from the previous and next models as a penalty term to ensure the stability of models at different times. However, this method is too simplistic, and calculations based on Euclidean distance lack the ability to analyze global information. Therefore, this invention proposes a distribution-based knowledge distillation method. A knowledge matrix is ​​formed by calculating the dot product of the output vectors of the previous and next models, and the distributions of the knowledge matrices at the two times are compared using KL divergence. This result is used as a penalty term to constrain the consistency of the global distribution of the previous and next models. The specific formula is as follows:

[0065]

number

[0066] Here, in Equation (16), the symbol (16-1) denotes the i-th feature vector extracted in the t-th environment, and d is the dimension of symbol (16-1). This KL divergence method not only optimizes the knowledge transfer process, allowing new models to more effectively learn knowledge from old models, but also improves the model's stability and generalization ability in multi-task learning environments. Furthermore, by adjusting the proportion of the KL divergence loss in the total loss function, the relationship between learning new knowledge and retaining old knowledge can be flexibly adjusted, further improving the scene recognition performance of lifelong learning robots in complex environments.

[0067] To achieve lifelong learning and avoid memory overload, models typically use a memory recall method, storing images of past scenes for learning, achieving long-term learning. In this invention, a biology-based learning process is designed to mimic the human memory system and realize lifelong learning. At the start, the video information stored by the robot is divided into images, which are then queued in a first-in, first-out order and regarded as the robot's short-term memory module.

[0068] When the short-term memory module becomes full, the oldest image is stored in the panoramic memory module. As the number of images input increases, the memory area of ​​the panoramic memory module becomes full. Therefore, the ratio of the memory area of ​​the panoramic memory module to the amount of data read is set as the update probability. When an image is received from the short-term memory module, images greater than the update probability are not updated, and images smaller than the update probability are stored in a random position in the panoramic memory module. When the robot enters a completely new environment and begins learning, the panoramic memory module of the past environment is stored in the long-term memory module according to the following policy:

[0069]

number

[0070] Here, symbol (17-1) is the long-term memory module at the t-th task, symbol (17-2) is the panoramic memory module at the t-th task, and λ is the hyperparameter to be set.

[0071] The triple memory module successfully stores the memories of the three aging periods in separate spaces, and by retrieving images from all memory modules during learning, image diversity is ensured and learning efficiency is improved.In addition, knowledge distillation can be performed by using only the long-term memory module, and knowledge distillation can be performed using only the knowledge learned in both the previous and subsequent models.

[0072] Finally, the overall loss function obtained by combining the above modules is shown below.

[0073]

number

[0074] When comprehensively considering knowledge distillation and memory sensing mechanisms, we introduce a method that combines Kullback-Leibler divergence and knowledge distillation (KD) to optimize the model learning process in a lifelong learning framework and mitigate the catastrophic forgetting phenomenon that occurs when the model faces continuous learning tasks.

[0075] The Relational Memory-Perception Synapse (RMAS) policy constructs a 3*3 similarity matrix based on the three sampled groups in each learning step, which reflects the similarity between the anchor data, the positive sample data, and the two negative sample data after network processing. The Frobenius norm of this matrix is ​​calculated, and the partial derivatives of the parameters in the model are taken to estimate the importance of the model parameters in the current learning process. This information is used to protect important parameters in future learning and prevent the loss of important knowledge.

[0076] In addition, to effectively reduce forgetting during lifelong learning, the distribution-based knowledge distillation technique is adopted. This method focuses on the knowledge transfer between two successive learning steps of the model, and calculates the KL divergence of the model output probability distributions of the two steps, thereby accurately measuring from a global perspective and minimizing information loss, ensuring the ability of lifelong learning.

[0077] Step S5: The triple memory module is used to update the long-term memory module. As shown in Figure 2, the oldest scene in the long-term memory module is written to the panoramic memory module. When the space in the panoramic memory module is full, the data in the panoramic memory module is randomly swapped, with the ratio of the space capacity to the amount of data loaded as the update probability, so that the panoramic memory module can represent the current complete environment. After entering a new environment, the panoramic memory module of the previous environment is written proportionally to the long-term memory module, so that the long-term memory module can represent multiple environments up to the present.

[0078] Step S6: Steps S1 to S5 are repeated to complete the learning of the scene recognition network.

[0079] Step S7: The scene image is input to the scene recognition network, and a feature vector f' is output.

[0080] Step S8: Compare f' with the existing scene feature vectors in the database, and find the scene with the highest similarity as the recognition result, thereby realizing the scene recognition function.

[0081] Experimental results For scene recognition ability, we adopted the recall rate with 100% accuracy as the basic evaluation index. For lifelong learning ability of multiple scenes, we used t*t evaluation matrices P for a recognition task with t environments. t*t Construct one P i,jindicates the recognition ability of the scene in the jth environment after the system has learned the ith environment, and the evaluation metric is the recall at 100% precision mentioned above. The final average performance metric is obtained based on the evaluation matrix.

[0082]

number

[0083] The experimental results were evaluated on Nordland, a dataset with multiple seasonal environments, and Oxford Robotcar, a dataset with multiple lighting conditions.

[0084] [Table 1]

[0085] Figures 3-4 are images recorded during the course of the experiment.

[0086] It should be noted that the above is a preferred embodiment of the present invention, and those skilled in the art may make some improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A scene recognition method for a robot based on lifelong learning, comprising the steps of: Step S1: Acquire a scene image captured by a robot camera; Step S2: Preprocessing is performed on the scene image. Step S3: Input the preprocessed scene image into the long-term and short-term triple memory module, and randomly select one anchor data I from the long-term and short-term triple memory module. ａ is sampled as a training scene, and then training scene I ａ and multiple positive sample data I belonging to the same scene. ｐｉ are randomly sampled as learning samples, and the learning scene I ａ and multiple negative sample data I belonging to different scenes. ｎｉ Randomly sample the Learning Scene I ａ , the plurality of positive sample data I ｐｉ , the plurality of negative sample data I ｎｉ Construct a small sample set from Step S4: Input the data in the small sample set into the scene recognition network, and generate a small sample set f of corresponding feature vectors. ａ ,f ｐｉ ,f ｎｉ The cosine similarity between the feature vectors is used to determine the least similar feature vector f ｐ The feature vector f of a different scene that is most similar to ｎ and find the feature vector f ａ and one deep mining group consisting of three groups, Step S5: Perform storage update on the long-term and short-term triple storage module; Step S6: Repeat steps S1 to S5 to complete the learning of the scene recognition network. Step S7: Input the scene image to the scene recognition network and output the feature vector f'. Step S8: Compare the feature vector f' with existing scene feature vectors in the database, find the scene with the highest similarity as the recognition result, and realize the scene recognition function. A scene recognition method for a robot based on lifelong learning, characterized by:

2. The long-term and short-term triple memory module includes a short-term memory module, a panoramic memory module, and a long-term memory module; The short-term memory module is implemented using a first-in-first-out image sequence, The panoramic memory module is formed by the short-term memory module outputting the image and performing internal filtering. The method of claim 1 , wherein the long-term memory module is obtained by filtering a panoramic memory module of a historical scene.

3. The method of claim 1 , wherein the scene recognition network includes a VGG feature extractor and a VLAD aggregator.

4. The VGG feature extractor uses a VGG16 feature extractor; The VLAD aggregator uses a NetVLAD aggregator, The method of claim 3 , wherein the NetVLAD aggregator specifically achieves feature aggregation through clustering and residual summation.

5. The loss function of the scene recognition network is: [Equation 1] where t denotes the loss function for the tth environment, and λ １ , λ ２ is a hyperparameter, and L ｈａｒｄ－ｔｒｉｐｌｅｔ are the loss functions of the three groups of deep mining, and L ＲＭＡＳ is the relational memory-perception synaptic loss function, and L ＫＤ is the distribution-based knowledge distillation loss function, The loss function L of the three groups of deep mining ｈａｒｄ－ｔｒｉｐｌｅｔ Specifically, it is as follows: [Equation 2] Here, symbol (2-1) is the cosine similarity between the i-th negative sample image feature output by the scene recognition network and the anchor image feature, symbol (2-2) is the similarity between the j-th positive sample image feature output by the scene recognition network and the anchor image feature, and δ is an artificially set hyperparameter. The relational memory-perception synaptic loss function L ＲＭＡＳ Specifically, it is as follows: [Equation 3] Here, symbol (3-1) denotes the i-th parameter when the scene recognition network learns the t-th task, and N ｔ denotes the number of samples in the training batch, the symbol (4-1) denotes the Gram matrix consisting of three groups of samples when training up to task t, F is the Frobenius norm, The knowledge distillation loss function L based on the distribution ＫＤ Specifically, it is as follows: [Equation 4] Here, in equation (6), the symbol (6-1) indicates the i-th feature vector extracted by the scene recognition network in the t-th environment, and d is the dimension of the symbol (6-1).

6. The method of claim 1, characterized in that in step S5, specifically, the oldest scene in the short-term memory module is written to the panoramic memory module; when the space of the panoramic memory module is full, the data in the panoramic memory module is randomly swapped, using the ratio of the space capacity to the amount of data read as the update probability, to ensure that the panoramic memory module can represent the current complete environment; after entering a new environment, the panoramic memory module of the previous environment is written to the long-term memory module according to the ratio, so that the long-term memory module can represent multiple environments up to the present.

7. A computer-readable storage medium, A computer-readable storage medium having a computer program stored therein, the computer program causing the computer to carry out the method according to any one of claims 1 to 6 when the computer program is executed in the computer.

8. 1. A computer device comprising: a memory and a processor, A computer apparatus, characterized in that it stores executable code in said memory, and that when said processor executes said executable code, it performs the method of any one of claims 1 to 6.

Images