Continuous visual retrieval method and system based on feature refinement and feature correction

Abstract

The invention discloses a continuous visual retrieval method and system based on feature refinement and feature correction, and belongs to the technical field of visual retrieval. The method comprises the steps that a visual retrieval model is constructed, the visual retrieval model comprises a backbone network, a feature refining module, a feature correction module, an embedded layer and a classifier, the feature refining module is used for revealing important areas in a feature map M output by the backbone network, and the feature correction module is used for correcting the feature map M output by the backbone network; the feature correction module is used for suppressing semantics overlapped with old categories in new categories when the visual retrieval model is trained; constructing a training data set of the tth stage, and training the visual retrieval model ft on the training data set of the tth stage to obtain a visual retrieval model ft + 1; and obtaining a visual retrieval result of the target image based on the trained visual retrieval model ft + 1. According to the method, the forgetting problem in a continuous visual retrieval task can be solved, and the sustainable expansion capability of a visual retrieval model in a dynamically changing data environment is improved.

Description

Technical Field

[0001] The present invention relates to the field of visual retrieval technology, and in particular to a continuous visual retrieval method and system based on feature refinement and feature correction. Background Art

[0002] With the explosive growth of Internet data, people generate a large amount of multimedia data in daily life, mainly text, images, and videos. Among them, visual data such as images and videos contain rich visual information, and their amount is increasing rapidly. In order to efficiently and accurately retrieve content that meets user query needs from massive visual data, visual retrieval technology has become a research hotspot in industry and academia. Specifically, visual retrieval technology aims to search and return entries similar to query data from large-scale visual databases. At present, visual retrieval technology has been widely used in practical scenarios such as smart cities, smart e-commerce, smart medical care, and copyright protection. However, with the rapid growth of visual data, the sustainable expansion capability of visual retrieval technology has become increasingly important in practical applications. For example, in smart city scenarios, as the number of surveillance cameras increases, the number of surveillance targets involved is also increasing, which requires pedestrian and vehicle re-identification systems to be able to model new targets in real time or near real time and achieve accurate retrieval to quickly respond to security incidents. In smart e-commerce business, a large number of emerging products are flooding into the market every day. If e-commerce software cannot update its search system in time to support the retrieval of emerging products, user experience will be seriously affected. Therefore, considering the dynamic changes of data in actual application scenarios, it is of great practical significance to develop a continuous visual retrieval technology that can quickly model and retrieve new categories. At present, continuous visual retrieval technology has proposed a series of learning techniques to ensure that the model does not forget old knowledge while learning new category knowledge, thereby improving retrieval accuracy and model update efficiency. For example, some methods introduce the maximum mean discrepancy (MMD) loss to minimize the difference between the embeddings generated by the new and old models. Some methods propose the backward compatibility requirement of features to ensure that the newly generated feature embeddings are comparable to the embeddings previously saved in the database.

[0003] Although existing methods take into account the backward compatibility between new features and features saved in the database, they can update the image retrieval system without changing the features of the saved database. However, these methods often ignore the in-depth study of the "semantic drift" problem. Over time, they often encounter the problem that the new features are too far away from the saved features, resulting in a decrease in retrieval accuracy. Through in-depth analysis, researchers found that one of the main causes of the "semantic drift" problem is the overlapping semantics between the new and old categories. Specifically, samples of new and old categories may share certain semantic features. For example, the new category "dog" may have similarities with the old category "cat" in terms of fur color, background, etc. During the model update process, since the training data of the new category samples dominates, the model will be biased towards the new category. This may cause the model to mistakenly bind these overlapping semantic features, such as fur color and background, to the new category "dog", while the old category "cat" containing overlapping semantics will be mistakenly identified as the new category, resulting in the forgetting of the old category, limiting the sustainable expansion ability of the model. To solve this problem, it is necessary to develop algorithms that can identify and process overlapping semantics between new categories in order to reduce semantic drift during model updates and ensure the accuracy and scalability of visual retrieval systems in continuous learning. Summary of the invention

[0004] To solve the above problems, the present invention discloses a continuous visual retrieval method and system based on feature refinement and feature correction, which are used to solve the forgetting problem in continuous visual retrieval tasks and enhance the sustainable expansion capability of the visual retrieval model in the face of dynamically changing data environments.

[0005] To achieve the above objectives, the technical solution of the present invention includes the following contents.

[0006] A continuous visual retrieval method based on feature refinement and feature correction, the method comprising:

[0007] Constructing a visual retrieval model, the visual retrieval model comprising: a backbone network, a feature refinement module, a feature correction module, an embedding layer and a classifier, the feature refinement module is used to reveal important areas in the feature map M output by the backbone network, and the feature correction module is used to suppress the semantics of new categories that overlap with old categories when training the visual retrieval model;

[0008] Construct the training dataset of the tth stage and train the visual retrieval model f on the training dataset of the tth stage t , and obtain the visual retrieval model f t+1 ; Wherein, the training data set of the tth stage includes: the new data appearing in the tth stage and the replay data in the previous t-1 stages, t is a positive integer;

[0009] Based on the trained visual retrieval model ft+1 Get the visual retrieval results for the target image.

[0010] Furthermore, in the case of t=1, the visual retrieval model f is trained on the training data set of the tth stage. t , and obtain the visual retrieval model f t+1 ,include:

[0011] Based on the backbone network of the visual retrieval model f1, the data x in the training dataset of the first stage is generated i The characteristic map M i , i is a natural number, the data x i This is the new data that appeared in the first stage;

[0012] The characteristic spectrum M i Input the feature refinement module to obtain the refined feature map M′ i ;

[0013] The refined feature map M′ is calculated via the embedding layer i The feature embedding f1(x i ), and embed the features into f1(x i ) Input the classifier to calculate the intra-stage discriminative loss

[0014] Based on intra-stage discriminative loss The backbone network, embedding layer, and classifier are learned end-to-end to obtain the visual retrieval model f2.

[0015] Furthermore, when t>1, the visual retrieval model f is trained on the training dataset of the tth stage. t , and obtain the visual retrieval model f t+1 ,include:

[0016] Based on the visual retrieval model t The backbone network generates the data x in the training data set of the tth stage i The characteristic map M i , i is a natural number, the data x i is the new data appearing in the tth stage or the replayed data in the previous t-1 stages;

[0017] The characteristic spectrum M i Input the feature refinement module to obtain the refined feature map M′ i ;

[0018] Calculate the refined feature map M′ i The feature embedding f t (x i ), and embed the features into ft (x i ) Input the classifier to calculate the intra-stage discriminative loss

[0019] Calculate the inter-stage data consistency loss based on the replayed data in the previous t-1 stages To ensure that the visual retrieval model trained in the tth stage is consistent with the visual retrieval in the previous t-1 stages;

[0020] Construct sample pairs (x a ,x n ), and based on the sample pair (x a ,x n ) Calculate the consistency loss of adjacent stages To transfer similarity knowledge from the visual retrieval model f t-1 Passed to the visual retrieval model f t ; where x a is the anchor point sample, x n is the anchor sample x a Difficult negative samples;

[0021] Based on old data from the previous t-1 phases The corresponding refined feature map Refine the feature map M′ i Correction is performed and based on the correction feature map Calculate feature correction loss and feature consistency loss Among them, the feature correction loss To maximize the correction feature map Image embedding generated after passing through the embedding layer With image x i Corresponding classification weight The similarity between the two, the feature consistency loss To correct the characteristic map As the goal, minimize the refined feature map M′ i and corrected feature maps The l2 distance between them;

[0022] Based on intra-stage discriminative loss Data consistency loss between stages Neighbor-stage consistency loss Feature Correction Loss And feature consistency loss The backbone network, embedding layer, and classifier are learned end-to-end to obtain the visual retrieval model f t+1 .

[0023] Further, the characteristic spectrum M i Input the feature refinement module to obtain the refined feature map M′ i ,include:

[0024] Calculate the feature map M on the channel dimension of the feature map i The average feature map A;

[0025] Calculate the mean of the average feature map A in the height and width dimensions of the feature map, and use the mean as the threshold τ for determining the importance of the region;

[0026] According to the threshold τ, calculate the feature map M i The importance weight matrix W;

[0027] The characteristic spectrum M i And the importance weight matrix W element-by-element dot multiplication, to obtain the refined feature map M′ i .

[0028] Furthermore, the discriminative loss in the stage in, Represents data x i Category y i The l2-normalized representation of the corresponding classification layer weights, κ represents the temperature coefficient, C 1:t Represents the set of known categories from the 1st stage to the tth stage.

[0029] Furthermore, the data consistency loss between the stages Among them, C 1:t-1 represents the set of known categories from the 1st stage to the t-1th stage, express, represents the replay data belonging to the known category c, ξ c Represents the prototype embedding of the known category c.

[0030] Furthermore, the adjacent stage consistency loss Among them, G t represents the database image that appears in the tth stage, m represents the threshold.

[0031] Furthermore, based on the old data in the previous t-1 stages The corresponding refined feature map Refine the feature map M′ i Correction, including:

[0032] Calculating old data Feature Embedding

[0033] By computing the feature embedding and the feature embedding f t (x i ), select the old data

[0034] For selected old data Calculate the old data Refined feature map

[0035] Using refined feature maps Refine the feature map M′ i Correction is performed to obtain the corrected feature map Among them, μ is a hyperparameter that controls the correction strength.

[0036] Furthermore, the feature correction loss The feature consistency loss in, Represents data x i Category y i The l2-normalized representation of the corresponding classification layer weights, Represents the corrected feature map The feature embedding generated after the embedding layer, κ represents the temperature coefficient, C 1:t Represents the known category set from the 1st stage to the tth stage, C, H, W are divided into feature maps M i The number of channels, height, and width.

[0037] A continuous visual retrieval system based on feature refinement and feature correction, the system comprising:

[0038] A model building module, which builds a visual retrieval model, wherein the visual retrieval model includes: a backbone network, a feature refinement module, a feature correction module, an embedding layer and a classifier, wherein the feature refinement module is used to reveal important areas in the feature map M output by the backbone network, and the feature correction module is used to suppress the semantics of new categories that overlap with old categories when training the visual retrieval model;

[0039] Model training module, used to construct the training dataset of the tth stage and train the visual retrieval model f on the training dataset of the tth stage t , and obtain the visual retrieval model f t+1 ; Wherein, the training data set of the tth stage includes: the new data appearing in the tth stage and the replay data in the previous t-1 stages, t is a positive integer;

[0040] Visual retrieval module, used to retrieve data based on the trained visual retrieval model f t+1Get the visual retrieval results for the target image.

[0041] Compared with the prior art, the present invention has at least the following beneficial effects.

[0042] The present invention designs a feature refinement module, which aims to enhance the model's ability to extract features with category discrimination, while reducing the impact of noise and background semantics in the image on the retrieval effect, and effectively alleviates the semantic overlap problem between new and old category features. Furthermore, the present invention proposes a feature correction module, which realizes the correction of the new category feature map by comparing the new and old category feature maps. In this way, the present invention can ensure that the model can learn the features of the new category without forgetting the knowledge of the old category during the continuous learning process. Experimental results show that the method of the present invention can significantly improve the retrieval performance of the model in the continuous learning scenario. This not only confirms the effectiveness of feature refinement and feature correction techniques, but also demonstrates the potential of the present invention in improving the sustainable expansion capabilities of visual retrieval systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 Framework diagram of the FRC-CVS method.

[0044] Figure 2 Recall@1 of different methods at different stages.

[0045] Figure 3 Sensitivity experiments on hyperparameters μ and γ.

[0046] Figure 4 Visualize the results. DETAILED DESCRIPTION

[0047] The present invention is further described in detail below in conjunction with specific embodiments. The given examples are only for illustrating the present invention, but not for limiting the scope of the present invention.

[0048] like Figure 1As shown, the continuous visual retrieval method based on feature refinement and feature correction (Feature Refinement and Calibration for Continual Visual Search) proposed in the present invention is mainly divided into three parts. The first part is a visual retrieval continuous learner, which is used to extract image features with sustainable expansion capabilities. The second part is a feature refinement module, which generates a weight matrix by aggregating the feature maps generated by the network. Using this weight matrix, the model can suppress category-irrelevant information such as noise and background in the picture, while enhancing the discriminative information related to the category. The third part is a feature correction module, which identifies and suppresses the semantics of the new category feature map that overlaps with the old category by comparing the feature maps of the new and old categories. In this way, the model is guided to explore and learn the unique discriminative information of the new category, thereby reducing the semantic overlap between the new and old categories.

[0049] 1. Background knowledge:

[0050] In a continuous learning scenario with T stages, the present invention uses G t represents the database image that appears in the tth stage (t = 1, 2, ..., T). In stage t, the present invention uses a deep model f t The image G t Mapping to feature embedding Where |G t | represents the number of database images, and d represents the embedding dimension. The database embedding g generated at the current stage t Then added to the database embedding collection In. Database embedded collection g 1:t is formed by the accumulation of database embeddings in the first t stages and is retained for the retrieval task in the subsequent stages {t+1,...,T}. Specifically, in the initial stage t=1, the trained model f1 maps the database image G1 to the embedding g1. In the continuous learning stage t (t>1), the deep model f t From t-1 Initialize, and continue training to transform G t Mapping to g t After stage t training, the new data is embedded in g t Added to the previously saved database embedding collection g 1:t-1 In the above example, the updated database embedding set g is formed. 1:t =g 1:t-1 ∪g t Therefore, the updated model f t Not only can you learn G t The newly appeared data in the database is compatible with the previously saved database embedding, so that ft The generated query is embedded in g 1:t In order to alleviate forgetting in the continuous learning scenario, the present invention adopts a data replay strategy. Specifically, after each stage t, a small number of images are added to the memory buffer, which can be replayed in subsequent sessions for training. The size of the memory buffer is fixed throughout the continuous learning stage.

[0051] (II) Continuous Learner for Visual Search (CVS):

[0052] The present invention adopts a sustainable learner for visual retrieval to learn image features with sustainable expansion capabilities. This sustainable learner contains three loss functions: inter-stage data consistency loss Consistency loss of adjacent stage models and intra-stage discriminative loss

[0053] Specifically, the inter-stage data consistency loss uses the replay data saved in the previous stage to ensure that the model f t The present invention first calculates all known categories c∈C 1:t-1 The average embedding of is taken as the prototype embedding ξ of this category c :

[0054] ξ c =E(g 1:t-1 (c)),c∈C 1:t-1

[0055] Among them, C 1:t-1 represents all known categories in the first t-1 stages, g 1:t-1 (c) is defined as the set of database feature embeddings of known category c in the first t-1 stages. Operation E(·) represents the expected operation. After that, the present invention calculates the data consistency loss between stages as follows:

[0056]

[0057] in Indicates playback data belonging to category c.

[0058] The consistency loss of adjacent stage models uses a triplet loss to transfer similarity knowledge from the model f of the previous stage t-1 The model f passed to the current stage t The present invention first constructs a sample pair (x a ,x n ), where x ais the anchor point sample, x n is the difficult negative sample of the anchor sample. After that, the consistency loss of adjacent stages is calculated as follows:

[0059]

[0060] in, m is the threshold (default is 0.1).

[0061] The intra-stage discriminative loss learns the discriminative embedding of categories in the form of classification and is calculated as follows:

[0062]

[0063] Among them, x i The database image G from the current stage t and the replay data from the previous stage, Represents category y i The l2-normalized representation of the corresponding classification layer weights, κ represents the temperature coefficient, and the default setting is 0.05.

[0064] In summary, the loss of the visual retrieval sustainable learner is:

[0065]

[0066] Among them, α is set to 10 and β is set to 1.

[0067] (III) Feature Refinement module (FR):

[0068] This paper proposes a feature refinement module to learn more fine-grained class discriminative information. First, the module uses aggregated feature maps (such as the output of the last convolutional layer of ResNet18) to reveal the important areas used by the model to identify the corresponding categories. These feature maps contain rich high-dimensional information and are crucial for capturing discriminative information in images. This paper defines is the feature map of the input image x, where C, H, and W represent the number of channels, height, and width of the feature map, respectively. In order to reveal the discriminative regions in the feature map, the present invention first calculates the average feature map A in the channel dimension:

[0069]

[0070] Then the present invention calculates the mean of the average feature map A and uses it as the threshold τ for determining the importance of the region:

[0071]

[0072] Among them, A h,wrepresents the element in the hth row and wth column of the average feature map A. According to the threshold τ, the present invention calculates the importance weight matrix W of the feature map:

[0073]

[0074] Among them, W h,w Represents the element in the hth row and wth column of the weight matrix W. By using the weight matrix W, the present invention can reveal the important areas in the feature map M, while suppressing irrelevant semantics such as noise and background in the image, thereby highlighting the targets related to the category. The refined feature map M ′ The calculation is as follows:

[0075] M ′ =M⊙W

[0076] where ⊙ is an element-wise multiplication.

[0077] (IV) Feature Calibration module (FC):

[0078] This paper proposes a feature correction module, which aims to alleviate the feature drift problem by suppressing the semantics of new categories that may overlap with old categories. Specifically, this paper first analyzes the similarity of features of new and old category samples to identify new category samples that may contain overlapping semantics. For example, a new category sample x in a given batch i , the present invention extracts its feature embedding as f t (x i ), after which the present invention identifies old category samples in the batch Its feature embedding With f t (x i ) is higher than other old category samples in the batch. Then, the present invention uses M′ i and Respectively represent x i and The refined feature map is used For M′ i After correction, the corrected feature map is:

[0079]

[0080] Among them, μ is a hyperparameter that controls the correction strength. The present invention further proposes a feature correction loss To guide the model to learn more class discriminative information. Specifically, the feature correction loss will maximize the correction feature map Image embedding generated after passing through the embedding layer With image x i Corresponding classification weight The similarity between:

[0081]

[0082] At the same time, the present invention proposes a feature consistency loss To encourage the model to preferentially extract features from the rectified feature map from new category images. Specifically, the feature consistency loss converts the rectified feature map As the goal, minimize M′ i and The l2 distance between:

[0083]

[0084] In summary, the loss function of the continuous visual retrieval method based on feature refinement and feature correction proposed in the present invention is:

[0085]

[0086] Among them, γ is a hyperparameter used to balance different losses. In the initial stage, that is, when t=1, the present invention only uses the intra-stage discriminative loss. The deep model f1 (including the backbone network, embedding layer and classifier) ​​is learned end-to-end. In the subsequent stage, that is, when t>1, the present invention uses the loss L of the visual retrieval sustainable learner cvs And the loss in the feature refinement and correction module and For the deep model f t (including backbone network, embedding layer and classifier) ​​for end-to-end learning.

[0087] Next, a specific experiment is used to illustrate the feature refinement and feature correction method for continuous visual retrieval provided by the present invention.

[0088] 1. Dataset:

[0089] This experiment selects three commonly used datasets in the field of image retrieval to verify the effectiveness of the proposed method, namely CIFAR10, CIFAR100 and Tiny ImageNet. CIFAR10 contains a total of 60,000 pictures belonging to 10 categories. This experiment randomly samples 1,000 pictures from all pictures (100 pictures for each category) as a query set, and the remaining pictures constitute the database set. In addition, this experiment randomly sampled 5,000 pictures (500 pictures for each category) from the database set as a training set. CIFAR100 contains a total of 60,000 pictures belonging to 100 categories. This experiment randomly sampled 10,000 pictures (100 pictures for each category) from all pictures as a query set, and the remaining pictures constitute the database set. At the same time, this experiment uses the database set as a training set to train the hash model of the present invention. The Tiny ImageNet dataset contains 120,000 images belonging to 200 categories, of which the training set has 100,000 images (500 images per category), the validation set has 10,000 images (50 images per category), and the test set has 10,000 images (50 images per category). In this experimental setting, the validation set is used as the query set, and the training set is used not only for model training but also as a database set.

[0090] 2. Evaluation indicators:

[0091] This experiment uses recall and average recall (AR) as evaluation indicators, where Recall@k represents the proportion of relevant images retrieved in the first k search results. AR@k represents the average value of Recall@k obtained in all retrieval stages in the continuous learning task. This indicator can more comprehensively reflect the overall retrieval performance of the model after multiple learning stages, rather than just the performance in a single stage. The present invention sets k to 1, 2, or 4.

[0092] 3. Target model:

[0093] In the experiment, the present invention uses ResNet18 as the backbone network, and the feature embedding of the image extracted by the network is 128 dimensions. This network is selected for its excellent feature extraction ability and wide applicability in various visual tasks. In each stage of continuous learning, this experiment trains the model for 100 epochs on the CIFAR10 dataset and 256 epochs on the CIFAR100 and TinyImageNet datasets. During the training process, this experiment sets the batch size to 64 images. This experiment uses the SGD optimizer with an initial learning rate of 0.03, and uses the cosine learning rate scheduler to dynamically adjust the learning rate within the range of 0.03 and 0.0003. At the end of each stage, this experiment saves a small part of the training images in the buffer and replays them during the next stage of training. This experiment sets the buffer size to 500 images on the CIFAR10 dataset, 2,000 images on the CIFAR100 dataset, and 4,000 images on the Tiny ImageNet dataset.

[0094] 4. Experimental results:

[0095] In order to verify the effectiveness of the invention, this experiment compares it with the current advanced continuous learning methods for image retrieval, including LwF, EWC, RWalk, MMD, BCT, Rainbow and CVS. Table 1 shows the average recall rate AR@k (k=1, 2, 4) after all stages of continuous learning. In addition, in order to more intuitively show the retrieval performance of the model after each stage, this experiment draws the curve of the recall rate Recall@1, as shown in Table 1. Figure 2 As shown. Based on the experimental results, the following conclusions can be drawn: (1) The experimental results in Table 1 show that the method of the present invention achieves the best results on AR@k (k = 1, 2, 4), significantly surpassing the previous best performing CVS method. Specifically, on the CIFAR10, CIFAR100 and TinyImageNet datasets, the method of the present invention achieved improvements of 5.67%, 2.37% and 2.81% respectively, indicating that the method proposed in the present invention can improve the continuous learning ability of the image retrieval model. (2) Figure 2 The results in show that at each stage of continuous learning, the method of the present invention can achieve better retrieval performance than the baseline method. This shows that the method of the present invention can achieve a better balance between learning new types of knowledge and preserving old types of knowledge.

[0096]

[0097] Table 1 Retrieval results of different methods on three datasets

[0098] 5. Module ablation experiment:

[0099] In order to verify the effect of different modules in FRC-CVS on the sustainable learning ability of the model, a series of module ablation analysis experiments were also conducted. These experiments verified the effects of the feature refinement module (FR) and the feature calibration module (FC) separately and jointly by selectively removing them in FRC-CVS. The experimental results are shown in Table 2. The first row of results shows the effect of deleting FR and FC at the same time in FRC-CVS, the second row of results shows the effect of retaining FR and deleting FC, the third row shows the effect of deleting FR and retaining FC, and the fourth row shows the effect of retaining FR and FC at the same time, that is, FRC-CVS.

[0100] (1) Effectiveness of the Feature Refinement Module: Comparing the results in the second row with those in the first row in Table 2, we can observe that the use of the feature refinement module FR can improve the performance of the baseline method. These improvements are because the FR module effectively captures discriminative information that is more relevant to the category, which helps to better resist forgetting in the subsequent learning stage.

[0101] (2) Effectiveness of the feature correction module: Comparing the results of the third row with the first and second rows in Table 2, it can be observed that the use of the feature correction module FC achieves a greater improvement than the feature refinement module FR. This is because the FC module can avoid the influence of the overlapping semantics of the new category features with the old category and guide the model to learn more novel discriminative features for the new category. Therefore. FC helps to achieve a better balance between acquiring new knowledge and maintaining old knowledge for old categories, effectively avoiding the problem of semantic drift.

[0102] (3) Effectiveness of the two modules working together: It can be observed from the fourth row of Table 2 that the method of the present invention can achieve further improvement by combining the feature refinement module FR with the feature correction module FC, indicating that the two modules proposed in the present invention can promote each other and work together to improve the sustainable learning ability of the model.

[0103] The results of the ablation experiments clearly demonstrate the importance and effectiveness of the FR and FC modules in FRC-CVS. Not only do these two modules contribute independently to the performance improvement, but their joint use can also achieve a synergistic effect, further enhancing the robustness and adaptability of the model in a continuous learning environment. Through the joint action of these modules, FRC-CVS can effectively maintain the memory of old category knowledge while learning new categories, thereby significantly improving the model's continuous learning ability.

[0104]

[0105] Table 2 Ablation of different modules

[0106] 6. Hyperparameter sensitivity experiment: In order to verify the sensitivity of the proposed method to the hyperparameters μ and γ, different values ​​were set for the two and multiple experiments were conducted. In the experiment, the value range of the two was set within {0.01, 0.05, 0.1, 0.5, 1}. When adjusting one hyperparameter, this experiment kept the value of the other hyperparameter unchanged. The experimental results are shown in Figure 2. Figure 3 As shown. Figure 3 It can be seen from the figure that when the hyperparameters fluctuate within a large range, the method of the present invention can still achieve stable performance, which indicates that the method of the present invention is more robust to changes in hyperparameters.

[0107] 7. Visualization results: In order to intuitively show the model's focus areas for different categories of samples, this experiment Figure 4 The attention map produced by the model is visualized in Figure 4 As can be seen from the figure, (a) compared with CVS, the attention map generated by the proposed method shows higher accuracy, which verifies that the feature refinement module FR can suppress noise and background semantics and highlight category-related features. (b) compared with CVS, the new and old attention maps generated by the proposed method on the same sample are more consistent, indicating that the feature calibration module FC successfully alleviates the feature drift problem.

[0108] In summary, the present invention proposes an innovative continuous learning method for a visual retrieval system. This method deeply analyzes the "semantic drift" problem and finds that the semantic overlap between new and old categories is one of the main causes of this problem. In order to alleviate the adverse effects of overlapping semantics, the present invention proposes an innovative continuous visual retrieval method, which is based on feature refinement and feature correction technology. It can not only improve the model's ability to extract features with category discriminativeness, but also reduce the overlapping semantics between new and old category features, and achieve learning new category features without forgetting old category features, thereby improving the accuracy and scalability of the visual retrieval system.

[0109] In order to extract features with class relevance, the present invention designs a feature refinement module. This module extracts features with more class discrimination by enhancing important areas in the feature map. Afterwards, the present invention designs a feature correction module. This module corrects features by identifying and suppressing overlapping semantics in the feature map of new categories, effectively avoiding semantic overlap between categories.

[0110] Experimental results show that the proposed method achieves higher retrieval accuracy than the baseline method on three widely used benchmark datasets. Visualization results show that the attention map extracted by the proposed method is more refined, and the attention map before and after the model update is basically consistent, which shows that the proposed method can extract features with high category discrimination, which are more robust in continuous learning and can alleviate the "semantic drift" problem.

[0111] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology in this field to understand the content of the present invention and implement it, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A continuous visual retrieval method based on feature refinement and feature correction, characterized in that: The method comprises: Constructing a visual retrieval model, the visual retrieval model comprising: a backbone network, a feature refinement module, a feature correction module, an embedding layer and a classifier, the feature refinement module is used to reveal important areas in the feature map M output by the backbone network, and the feature correction module is used to suppress the semantics of new categories that overlap with old categories when training the visual retrieval model; Construct the training dataset of the tth stage and train the visual retrieval model f on the training dataset of the tth stage t , and obtain the visual retrieval model f t+1 ; Wherein, the training data set of the tth stage includes: the new data appearing in the tth stage and the replay data in the previous t-1 stages, t is a positive integer; Based on the trained visual retrieval model f t+1 Get the visual retrieval results for the target image.

2. The method according to claim 1, characterized in that In the case of t=1, the visual retrieval model f is trained on the training dataset in the tth stage t , and obtain the visual retrieval model f t+1 ,include: Based on the backbone network of the visual retrieval model f1, the data x in the training dataset of the first stage is generated i The characteristic map M i , i is a natural number, the data x i This is the new data that appeared in the first stage; The characteristic spectrum M i Input the feature refinement module to obtain the refined feature map M′ i ; The refined feature map M′ is calculated via the embedding layer i The feature embedding f1(x i ), and embed the features into f1(x i ) Input the classifier to calculate the intra-stage discriminative loss Based on intra-stage discriminative loss The backbone network, embedding layer, and classifier are learned end-to-end to obtain the visual retrieval model f2.

3. The method according to claim 1, characterized in that In the case of t>1, the visual retrieval model f is trained on the training dataset in the tth stage t , and obtain the visual retrieval model f t+1 ,include: Based on the visual retrieval model t The backbone network generates the data x in the training data set of the tth stage i The characteristic map M i , i is a natural number, the data x i is the new data appearing in the tth stage or the replayed data in the previous t-1 stages; The characteristic spectrum M i Input the feature refinement module to obtain the refined feature map M i ′ ; Calculate the refined feature map M′ i The feature embedding f t (x i ), and embed the features into f t (x i ) Input the classifier to calculate the intra-stage discriminative loss Calculate the inter-stage data consistency loss based on the replayed data in the previous t-1 stages To ensure that the visual retrieval model trained in the tth stage is consistent with the visual retrieval in the previous t-1 stages; Construct sample pairs (x a ,x n ), and based on the sample pair (x a ,x n ) Calculate the consistency loss of adjacent stages To transfer similarity knowledge from the visual retrieval model t-1 Passed to the visual retrieval model f t ; where x a is the anchor point sample, x n is the anchor sample x a Difficult negative samples; Based on old data from the previous t-1 phases The corresponding refined feature map Refine the feature map M′ i Correction is performed and based on the correction feature map Calculate feature rectification loss and feature consistency loss Among them, the feature correction loss To maximize the correction feature map Image embedding generated after passing through the embedding layer With image x i Corresponding classification weight The similarity between the two, the feature consistency loss To correct the characteristic map As the goal, minimize the refined feature map M i ′ and corrected feature maps The l2 distance between them; Based on intra-stage discriminative loss Data consistency loss between stages Neighbor-stage consistency loss Feature Correction Loss And feature consistency loss The backbone network, embedding layer, and classifier are learned end-to-end to obtain the visual retrieval model f t+1 .

4. The method according to claim 3, characterized in that The characteristic spectrum M i Input the feature refinement module to obtain the refined feature map M′ i ,include: Calculate the feature map M on the channel dimension of the feature map i The average feature map A; Calculate the mean of the average feature map A in the height and width dimensions of the feature map, and use the mean as the threshold τ for determining the importance of the region; According to the threshold τ, calculate the feature map M i The importance weight matrix W; The characteristic spectrum M i And the importance weight matrix W element-by-element dot multiplication, to obtain the refined feature map M′ i .

5. The method according to claim 3, characterized in that: The discriminative loss within the stage in, Represents data x i Category y i The l2-normalized representation of the corresponding classification layer weights, κ represents the temperature coefficient, C 1:t Represents the set of known categories from the 1st stage to the tth stage.

6. The method according to claim 3, characterized in that Data consistency loss between the stages Among them, C 1:t-1 represents the set of known categories from the 1st stage to the t-1th stage, express, represents the replay data belonging to the known category c, ξ c Represents the prototype embedding of the known category c.

7. The method according to claim 3, characterized in that The adjacent stage consistency loss Among them, G t represents the database image that appears in the tth stage, m represents the threshold.

8. The method according to claim 3, characterized in that Based on old data from the previous t-1 phases The corresponding refined feature map Refine the feature map M′ i Correction, including: Calculating old data Feature Embedding By computing the feature embedding and the feature embedding f t (x i ), select the old data For selected old data Calculate the old data Refined feature map Using refined feature maps Refine the feature map M′ i Correction is performed to obtain the corrected feature map Among them, μ is a hyperparameter that controls the correction strength.

9. The method according to claim 3, characterized in that: The feature correction loss The feature consistency loss in, Represents data x i Category y i The l2-normalized representation of the corresponding classification layer weights, Represents the corrected feature map The feature embedding generated after the embedding layer, κ represents the temperature coefficient, C 1:t Represents the known category set from the 1st stage to the tth stage, C, H, W are divided into feature maps M i The number of channels, height, and width.

10. A continuous visual retrieval system based on feature refinement and feature correction, characterized in that: The system comprises: A model building module, which builds a visual retrieval model, wherein the visual retrieval model includes: a backbone network, a feature refinement module, a feature correction module, an embedding layer and a classifier, wherein the feature refinement module is used to reveal important areas in the feature map M output by the backbone network, and the feature correction module is used to suppress the semantics of new categories that overlap with old categories when training the visual retrieval model; Model training module, used to construct the training dataset of the tth stage and train the visual retrieval model f on the training dataset of the tth stage t , and obtain the visual retrieval model f t+1 ; Wherein, the training data set of the tth stage includes: the new data appearing in the tth stage and the replay data in the previous t-1 stages, t is a positive integer; Visual retrieval module, used to retrieve data based on the trained visual retrieval model f t+1 Get the visual retrieval results for the target image.

Images