A continuous learning-oriented cross-modal knowledge decoupling sketch image retrieval method

Abstract

The application relates to the technical field of cross-modal retrieval and continuous learning, and discloses a continuous learning-oriented cross-modal knowledge decoupling sketch image retrieval method, which explicitly decouples shared semantic information and modality-specific expression information in sketches and images through modality-common guidance and modality-specific guidance, avoids knowledge interference in continuous learning, extracts modality-common features, modality-specific features and shared feature vectors by using old models and new models respectively, makes shared features of sketches and images close to similar text features based on a contrast learning loss, and constructs a shared knowledge alignment loss; uses old category prototypes, extracts cosine similarities of joint features extracted by new and old models and each prototype, constructs a cross-stage prototype loss through KL divergence, and combines a ternary loss and a classification loss to train the model. Through knowledge decoupling, text semantic alignment and prototype distillation, the method effectively alleviates the catastrophic forgetting in continuous learning and improves the cross-modal retrieval accuracy.

Description

Technical Field

[0001] This invention relates to the field of cross-modal retrieval and continuous learning technology, and in particular to a cross-modal knowledge decoupling sketch image retrieval method for continuous learning. Background Technology

[0002] With the continuous development of deep learning technology, cross-modal information processing has gradually become an important research direction in the field of computer vision. Among them, sketch-based image retrieval aims to retrieve semantically consistent real images from large-scale image databases using user-drawn abstract sketches. Due to its natural interaction and flexible expression, it has broad application prospects in fields such as industrial design, product retrieval, and intelligent creation. To improve the model's generalization ability to unknown categories, zero-shot sketch image retrieval further requires the model to be able to retrieve images of categories that did not appear during the training phase. Existing methods typically improve retrieval accuracy by constructing a shared embedding space to map sketches and images to a unified feature space and by achieving cross-modal alignment through metric learning.

[0003] However, in practical applications, new sketch and image categories are constantly emerging, and traditional methods trained on static data struggle to cope with this continuous change. Therefore, the continuous sketch image retrieval task has emerged, aiming to maintain the ability to retrieve existing categories while continuously introducing new categories or data. This task faces a dual challenge: the model not only needs to address the modal differences between sketches and images but also needs to solve the catastrophic forgetting problem brought about by continuous learning. Existing methods often employ knowledge distillation or data replay strategies to mitigate forgetting, but these methods typically fail to adequately distinguish between shared semantic information and modality-specific information across modalities, leading to interference between different types of knowledge and consequently affecting the model's overall performance on both new and old tasks.

[0004] In summary, during continuous learning, the cross-modal features of sketches and images not only exhibit significant distributional differences, but also require the model to simultaneously learn new knowledge and retain old knowledge as new tasks are continuously introduced. However, existing methods typically align these features within a unified feature space, failing to effectively distinguish between modality-shared and modality-specific information, leading to conflicts between the two types of knowledge during the learning process. Furthermore, the performance of low-level visual structural information and high-level semantic information varies considerably across different modalities; direct fusion or alignment can easily introduce noise, thereby reducing retrieval robustness. Therefore, achieving effective decoupling and stable alignment of cross-modal knowledge within a continuous learning framework is a pressing issue that needs to be addressed. Summary of the Invention

[0005] Therefore, the technical problem to be solved by the present invention is to overcome the catastrophic forgetting problem caused by the failure of cross-modal retrieval in continuous learning scenarios in the prior art, which leads to mutual interference of knowledge due to the inability to effectively distinguish between modal shared semantics and modal-specific information, and the lack of an effective mechanism for retaining old knowledge.

[0006] To address the aforementioned technical problems, this invention provides a cross-modal knowledge decoupling sketch image retrieval method for continuous learning, comprising: Obtain the sketch image to be matched and its category text description in the current learning stage, as well as multiple real images, and perform feature encoding and dimension reshaping respectively to obtain sketch visual features, text features and multiple sample visual features; Using the old model from the previous learning stage, obtain the old modality common features, old modality specific features, old shared feature vectors and old joint features corresponding to each visual feature; the visual features include sketch visual features and multiple sample visual features; By utilizing the new model in the current learning phase, we can obtain the new modality-common features, new modality-specific features, new shared feature vectors, and new joint features corresponding to each visual feature. Based on the InfoNCE loss function, the new shared feature vectors corresponding to each visual feature are made closer to the text features of their corresponding category text descriptions to construct a shared knowledge alignment loss. Obtain the old prototypes corresponding to each image category in the previous learning stage, and construct the cross-stage prototype loss based on the old joint features of sketch visual features and the cosine similarity of each old prototype, the new joint features of sketch visual features and the cosine similarity of each old prototype, and the KL divergence. The shared knowledge alignment loss, cross-stage prototype loss, triplet loss, and classification loss are weighted and summed to obtain the total loss function. The new model is then trained to obtain the model trained in the current learning stage. Based on the sketch image to be matched and its category text description, the target real image is obtained.

[0007] The cross-modal knowledge decoupling sketch image retrieval method for continuous learning provided in this application has the following beneficial effects: By explicitly decoupling cross-modal knowledge, distinguishing between modal common features and modal-specific information, the mutual interference between different types of knowledge is effectively reduced, and the discriminative ability of feature representation is improved. By using the contrastive learning loss InfoNCE, the sketch shared feature vector and the sample shared feature vector are brought closer to the text features of the same category in the embedding space, and pushed away from the text features of different categories, respectively. With the help of the semantic generalization ability of the text modality, a unified semantic reference benchmark is provided for cross-modal shared features, which effectively makes up for the alignment difficulties caused by the modal gap between the sketch and the real image, and enhances the discriminativeness and robustness of the shared features at the category level.

[0008] During continuous learning, only sketch prototypes and image prototypes are saved for each seen category after each stage, avoiding the need to store historical raw data. During training in a new stage, joint features of the sketch vision are extracted using both the old model from the previous stage and the current model. The cross-modal affinity distribution between these features and the old prototypes is calculated, and a relationship matrix is ​​constructed. Knowledge distillation is then performed using KL divergence to ensure consistency in the relative relationship structure between the new and old models. This mechanism effectively preserves the decision boundaries of old categories without relying on access to old data, improving the method's practicality in privacy-constrained scenarios. It also significantly alleviates the catastrophic forgetting problem in continuous learning, achieving a dynamic balance between new knowledge learning and the retention of old knowledge.

[0009] This invention explicitly separates shared semantic information and modality-specific expressive information in sketch and image modalities; it eliminates modal style differences through instance normalization and dynamically fuses original and normalized features using channel masks, enabling shared modal features to focus on consistent semantic discriminative cues across modalities; simultaneously, it employs independent lightweight multilayer perceptrons to model the unique brushstrokes, textures, and other details of sketches and real images respectively; this decoupling mechanism effectively avoids mutual interference between different types of knowledge during continuous learning, ensuring that the joint retrieval features possess both cross-modal semantic consistency and modality-specific fine-grained discriminative capabilities, significantly improving retrieval accuracy. Attached Figure Description

[0010] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein: Figure 1 A flowchart illustrating the steps of a sketch image retrieval method for decoupling cross-modal knowledge for continuous learning; Figure 2 A schematic diagram illustrating the principle of a cross-modal knowledge decoupling sketch image retrieval method for continuous learning. Detailed Implementation

[0011] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0012] Please see Figure 1 , Figure 1 The diagram shows the steps of the cross-modal knowledge decoupling sketch image retrieval method for continuous learning, including steps S101 to S106.

[0013] S101: Obtain the sketch image to be matched and its category text description for the current learning stage, as well as multiple real images, and perform feature encoding and dimensionality reshaping on each to obtain sketch visual features, text features, and multiple sample visual features, including: Based on the black pixels in the sketch image to be matched, the smallest anchor box containing the subject is obtained; after extracting the smallest anchor box containing the subject and scaling it to a preset size, mean and variance normalization is performed to obtain a preprocessed sketch; the preprocessed sketch is input into the first image encoder to obtain the first encoded feature; the first encoded feature dimension is reshaped into a three-dimensional feature map of a preset size to obtain the sketch visual features. For each real image, it is scaled to a preset size and then normalized for mean and variance to obtain a preprocessed real image; the preprocessed real image is input into a second image encoder to obtain a second encoded feature; the second encoded feature dimension is reshaped into a three-dimensional feature map of a preset size to obtain the sample visual feature; Based on a preset text template, the category text description is standardized into a text prompt; the text prompt is then input into a text encoder to obtain text features.

[0014] Furthermore, due to the sparse lines in the sketch, this invention employs a multi-head cross-attention module to encode the first feature. Semantic interaction with text features t to enhance the discriminative power of the features includes: The first encoded feature is linearly transformed and mapped to a query matrix. Mapping text features to a key matrix AND-value matrix Based on the cross-attention mechanism, attention features are obtained, represented as: ; After performing residual connections between the attention features and the sketch visual features, and then performing layer normalization, the enhanced first encoded feature is obtained, represented as: .

[0015] S102: Using the old model from the previous learning stage, obtain the old modality common features, old modality specific features, old shared feature vectors and old joint features corresponding to each visual feature; the visual features include sketch visual features and multiple sample visual features.

[0016] S103: Utilize the new model in the current learning stage to obtain new modality-shared features, new modality-specific features, new shared feature vectors, and new joint features corresponding to each visual feature.

[0017] Specifically, the sketch visual features and the visual features of each sample are input into the old model of the previous learning stage to gradually obtain the old modality common features, old modality specific features, old shared feature vectors and old joint features corresponding to each visual feature; the sketch visual features and the visual features of each sample are input into the new model of the current learning stage to gradually obtain the new modality common features, new modality specific features, new shared feature vectors and new joint features corresponding to each visual feature.

[0018] In this embodiment, both intermediate sketch visual features and sample visual features are considered as visual features. The old model from the previous learning stage and the new model in the current learning stage are both considered as models. The specific process of inputting visual features into the model to obtain joint features is described, including: Normalize the visual features to obtain normalized features; The visual features and normalized features are respectively input into the lightweight channel mask generation network to generate the corresponding visual channel weight mask and normalized channel weight mask. Based on the channel-wise multiplication fusion method, visual features and normalized features are weighted and fused using visual channel weight masks and normalized channel weight masks to obtain the modal common features of visual features. Based on the multilayer perceptron, the encoded features are mapped to modality-specific features; The shared feature vector is obtained by flattening the common features of the modalities and concatenating it with the modality-specific features along the channel dimension to obtain the joint features of the visual features; Among them, the visual features are either sketch visual features or sample visual features, and the corresponding coding features are either the first coding feature or the second coding feature. When the model is the old model from the previous learning stage, the old modality shared features, old modality-specific features, old shared feature vectors, and old joint features of the sketch visual features and the visual features of each sample are obtained. When the model is the new model in the current learning stage, the new modality shared features, new modality-specific features, new shared feature vectors, and new joint features of the sketch visual features and the visual features of each sample are obtained.

[0019] Specifically, the learning stage Visual features in Visual channel weight mask , represented as: ; Learning stage Normalization features in Normalized channel weight mask , represented as: ; Among them, image categories , hour Visual channel weight mask representing the visual features of a sketch. hour A visual channel weight mask representing the visual features of a sample; hour Normalized channel weight mask representing the visual features of a sketch. hour Normalized channel weight mask representing the visual features of a sample; hour Indicate the visual features of the sketch, hour Indicates the visual features of the sample; hour Normalized features representing the visual characteristics of a sketch hour Normalized features representing the visual features of a sample; This represents the Sigmoid activation function. and Learnable parameters representing a lightweight channel mask generation network for visual feature input. Represents the ReLU nonlinear activation function. Indicates global average pooling. and The learnable parameters represent the lightweight channel mask generation network with normalized feature inputs.

[0020] Specifically, the modal common features of visual features are represented as follows: ; in, This indicates multiplication by channel.

[0021] Specifically, modality-specific features are represented as follows: ; in, Indicates the learning stage Mid-visual features Modal-specific characteristics, Representing visual features The corresponding multilayer perceptron, Indicates the learning stage Image categories The coding features, Indicates a random deactivation layer. Represents the ReLU activation function layer; , , , For learnable model parameters, , , This represents the hidden layer dimension of a multilayer perceptron. The dimension representing the encoded feature.

[0022] S104: Based on the InfoNCE loss function, the new shared feature vectors corresponding to each visual feature are made closer to the text features of their corresponding category text descriptions to construct a shared knowledge alignment loss, expressed as: ; in, This represents the natural exponential function. Represents the cosine similarity function. Indicates the learning stage A new shared feature vector of visual features in sketches. Indicates the learning stage The middle category is The text features of the category text description This indicates the preset temperature parameter. Represents a set of categories. Indicates the learning stage A new shared feature vector of visual features of medium-sized samples.

[0023] This invention extracts category-specific text features and uses contrastive learning loss (InfoNCE) to bring sketch-shared feature vectors and sample-shared feature vectors closer to similar text features and further away from dissimilar text features in the embedding space. By leveraging the semantic generalization ability of text modalities, it provides a unified semantic reference benchmark for cross-modal shared features, effectively compensating for the alignment difficulties caused by the modal gap between sketches and images, and enhancing the discriminativeness and robustness of shared features at the category level.

[0024] S105: Obtain the old prototypes corresponding to each image category in the previous learning stage, based on the cosine similarity between the old joint features of the sketch visual features and each old prototype, and the cosine similarity between the new joint features of the sketch visual features and each old prototype. Using KL divergence, construct the cross-stage prototype loss, including: S105-1: Calculate the sketch image to be matched based on the cosine similarity between the sketch visual features in the old joint features of the old model and each old prototype. With the The old affinity of the Gejiu prototype is represented as: ; S105-2: Acquisition Learning Phase middle sketch image to be matched The old model affinity distribution matrix , represented as: ; in, Indicates based on learning stage Sketch images to be matched obtained from the old model The old characteristics of the union, Indicates the learning stage The first in An old prototype sample, Indicates temperature parameter; Indicates the learning stage The total number of old prototypes in the sample, Indicates the learning stage The first in One sample old prototype; Represents the row normalization function. Indicates transpose; S105-3: Calculate the sketch image to be matched based on the new joint features in the new model and the cosine similarity between the sketch visual features and each old prototype. With the The New Affinity of Gejiu Prototype , represented as: ; S105-4: Acquisition Learning Phase middle sketch image to be matched The affinity distribution matrix of the new model is expressed as: ; in, Indicates based on learning stage Sketch images to be matched obtained by the new model New joint features; S105-5: Using the affinity matrix of the old model as the distillation objective, the sketch image to be matched is minimized by KL divergence. The difference between the affinity distribution matrix of the new model and the affinity distribution matrix of the old model is used to construct a cross-stage prototype loss. , represented as: .

[0025] In this invention, during continuous learning, at the end of each stage, only two lightweight vectors—the sketch shared prototype and the image shared prototype—are saved as the old model for each seen category, thus avoiding the need to store historical raw data. During training in a new stage, joint features are extracted using both the old model from the previous stage and the current model. The cross-modal affinity distribution between these features and the old prototype is calculated, and a relationship matrix is ​​constructed. Knowledge distillation is then performed using KL divergence to ensure that the new and old models maintain consistency in the relative relationship structure between samples. This mechanism effectively preserves the decision boundaries of the old categories without relying on access to old data, significantly mitigating the catastrophic forgetting problem in continuous learning and achieving a dynamic balance between new knowledge learning and the retention of old knowledge.

[0026] Meanwhile, this invention only needs to save two shared prototype vectors (with dimensions consistent with shared features) and old model parameters for each class, resulting in extremely low storage overhead. This avoids the storage requirements of traditional data playback methods for original image data, making it particularly suitable for practical application scenarios where data privacy is limited, storage resources are scarce, or the data scale is massive.

[0027] S106: Weight the shared knowledge alignment loss, cross-stage prototype loss, triplet loss and classification loss to obtain the total loss function, train the new model, obtain the model trained in the current learning stage, and perform retrieval based on the sketch image to be matched and its category text description to obtain the target real image.

[0028] Specifically, this embodiment uses sketch visual features as anchor points, takes sample visual features of the same category as sketch visual features as positive samples, and takes sample visual features of different categories as negative samples, to construct a triplet loss. Using sketch visual features enhanced with textual semantics. As sketch anchors, sample visual features of real images of the same category As positive samples, the visual features of samples from real images of different categories As negative samples, the distance between sketch image pairs of the same class is made smaller than the distance between pairs of different classes, as shown in the formula: ; Where m is the interval parameter.

[0029] If the implementation does not enhance the sketch visual features, then the original sketch visual features are used to construct the triplet loss, which is expressed as: .

[0030] Specifically, this embodiment constructs a classification loss based on the true category of the sketch image to be matched in the current learning stage and the category predicted by the new model. ; Visual features of the sketch after semantic enhancement and sample visual features By inputting the weight-sharing classifier, the corresponding class prediction probability is obtained. and utilize real category labels The cross-entropy loss is calculated and expressed as: .

[0031] Based on the above embodiments, the cross-modal knowledge decoupling sketch image retrieval method for continuous learning proposed in this invention is applied to a practical continuous learning scenario, referring to... Figure 2The diagram illustrates the principle of a cross-modal knowledge decoupling sketch image retrieval method for continuous learning. Specifically, the data organization format in a continuous learning scenario is first defined. Assume the continuous learning process includes S sequentially arriving tasks. The training dataset is denoted as stage. ,in, The input is a sketch image. A real image corresponding to the sketch. For category labels, For the corresponding category text description, For the first The number of samples in each stage. When training the current stage, the model cannot access training data from previous stages; it can only learn and update its knowledge based on the data from the current stage. Based on this setting, continuous sketch retrieval is performed, with specific steps shown in S201 to S208.

[0032] S201: Preprocess the sketch, the real image, and the corresponding text description respectively.

[0033] For a real image, scaling it to a fixed size yields an image input in tensor form. ,in and The height and width of the image are defined; then mean and variance normalization is performed to eliminate irrelevant interference such as lighting and color, unify the image data distribution, and obtain a preprocessed real image.

[0034] For sketch data, hand-drawn sketches often contain a large amount of blank background, and directly extracting features will lead to sparse representation. Therefore, the minimum anchor box containing the main subject is determined based on the black pixels in the sketch, and the blank areas outside the anchor box are cropped, retaining only the core line structure. Then, the cropped sketch is scaled to the target size to obtain the final sketch. The system employs a normalization method consistent with that used for images to ensure the alignment of the distributions of the two types of visual data, thereby obtaining a preprocessed sketch.

[0035] For text data, the category text uses the category name as semantic guidance. For each category name, "class", the text template "a photo of a {class}" from the CLIP model is used as a text prompt to generate a standardized category description.

[0036] S202: Perform feature encoding on the sketch, real image, and text description respectively to obtain the corresponding modal features.

[0037] Because there are significant modal differences between the sketch domain and the real image domain, embodiments of the present invention assign an independent encoder to each modality. Specifically, the sketch encoder... With image encoder Both methods use Vision Transformer as the backbone network to extract visual features from preprocessed sketch samples and real image samples to obtain the first encoded features. With the second coding feature Meanwhile, for category text descriptions, a pre-trained CLIP text encoder is used. Extracting text features .

[0038] Because the sketch lines are sparse, this embodiment of the invention uses a multi-head cross-attention module to encode the first feature. Semantic interaction is performed with text features t to enhance the discriminative power of the features; the first encoded features are mapped to a query matrix Q through a linear transformation, and the text features are mapped to a key matrix K and a value matrix V respectively, using cross-attention. After residual connection and layer normalization, the enhanced first coding feature is obtained. .

[0039] For the extracted encoded features and This involves dimensional reshaping, transforming it into a three-dimensional feature map, i.e., sketch visual features. and sample visual features Where C represents the number of feature channels, and H and W correspond to the height and width of the feature map, respectively.

[0040] A natural modal gap exists between sketches and real images. Sketches have sparse lines, retaining only the outline and structural information of objects, while real images contain rich color, texture, and lighting details, resulting in significant differences in modal feature distribution. If only the discriminative features shared by both are learned, the model cannot fully utilize the fine-grained information of the image, leading to insufficient retrieval capabilities. If only the modality-specific information is learned, cross-modal matching is easily affected by style differences, resulting in decreased stability. In the context of continuous incremental learning, when the model learns new categories at each stage, the feature distribution of old and new categories is prone to shift, further amplifying feature conflicts caused by modal differences. Therefore, this invention decouples visual features, splitting the uniformly encoded visual information into modality-shared knowledge and modality-specific knowledge. The modality-shared knowledge is used to construct a stable semantic association between sketches and images. The modality-specific knowledge, on the other hand, mines visual cues unique to each modality, supplementing fine-grained discriminative capabilities.

[0041] S203: Modal shared feature extraction: In the modal shared guidance mechanism, instance normalization is performed on the visual features of the sketch and the real image to eliminate the interference caused by modal style differences.

[0042] In this embodiment, taking a sketch as an example, the modal common features of the sketch are extracted, including S203-1 to S203-3.

[0043] S203-1: Obtain the normalized features of the sketch , represented as: ; in, , These represent the mean and variance of each sketch's visual feature in the spatial dimension. To prevent division by zero for extremely small constants; this operation normalizes the feature map distribution of each sample, effectively suppressing the bias caused by the difference between sketch and image modalities.

[0044] S203-2: After normalization, two lightweight channel mask generation networks are set up based on the modal common guidance mechanism to process the visual features of the sketch. With normalized feature map Feature filtering includes: First, perform global average pooling on the visual features to compress the spatial dimension. Then, sequentially pass the features through linear transformation, ReLU activation function, and Sigmoid activation function to generate the corresponding channel weight mask. , represented as: ; Here, pool represents global average pooling operation. It is a ReLU nonlinear activation function. It is the Sigmoid activation function. All of these are learnable weight parameters of the network; Similarly, normalize the feature map By performing the same operation, global average pooling is applied to compress the spatial dimension. Then, the normalized feature map is obtained by sequentially applying linear transformation, ReLU activation function, and Sigmoid activation function to obtain the channel weight mask. , represented as: .

[0045] S203-3: Combining the two sets of channel weight masks, the modal common features of the s-th stage sketch are finally obtained through a channel-by-channel multiplication fusion method. The calculation process is as follows: ; Among them, symbols This represents element-wise multiplication operations along the channel dimension. Used to characterize the importance of representing modal common knowledge after eliminating modal differences; It is used to supplement discriminative information dynamically after suppressing modal differences, based on the importance of modal common knowledge in the original features.

[0046] Furthermore, the same operation is performed on the real image to obtain the normalized features of the real image. Channel weight mask and and modal common features .

[0047] S204: Modality-specific feature extraction.

[0048] While extracting modal common features, this invention further introduces a modality-specific guidance mechanism to capture the unique discriminative information of the sketch and the image, respectively. While modal common knowledge can support cross-modal semantic alignment, it loses detailed cues that exist only in a single modality (such as the brushstroke style of the sketch and the texture and color of the image), which are crucial for improving the discriminative power of instance-level retrieval. Ignoring modality-specific knowledge can easily lead to confusion between new and old knowledge from different modalities during continuous learning, exacerbating forgetting. Therefore, the modality-specific guidance mechanism employs two lightweight and independent multilayer perceptrons (MLPs), which respectively apply to the original instance features of the sketch and the real image, outputting modality-specific feature vectors.

[0049] Specifically, the modality-specific guidance module uses a sketch-specific perceptron. The first encoded feature corresponding to the sketch Original instance features are mapped to sketch modality-specific features. Similarly, the image side uses an image-specific perceptron. The second encoded feature is mapped to image modality-specific features. The calculation formula is as follows: ; ; in, , , , Learnable parameters for the sketch side. The hidden layer dimension is used. For images, the same calculation method is employed, with the same structure but not shared. Dropout layers are used to prevent overfitting, and the ReLU activation function introduces nonlinearity, allowing the unique information of each modality to be modeled in an independent parameter space, avoiding interference with shared knowledge.

[0050] S205: Constructing Joint Features: Taking a sketch as an example, due to the shared features of the modes It takes the form of a three-dimensional feature map, while possessing modal-specific features. Since the features are one-dimensional vectors, the shared modal features are first converted into vector form through a flattening operation to obtain the shared feature vector. Then, it is concatenated with the modality-specific features along the channel dimension to obtain the sketch joint features. .

[0051] Similarly, for real images, joint image features can be obtained. .

[0052] This invention explicitly decouples cross-modal knowledge, distinguishes between shared semantic information and modality-specific information, effectively reduces mutual interference between different types of knowledge, and improves the discriminative ability of feature representations.

[0053] S206: Constructing Shared Knowledge Alignment Loss: To further enhance the semantic consistency of shared features across modalities, this invention introduces a text modality as a semantic bridge to align the shared features between the sketch and the image. Using InfoNCE loss based on contrastive learning, the shared feature vectors of the sketch... Shared feature vectors with real images The shared knowledge alignment loss is constructed by aligning text features of the same category with those of different categories in the embedding space, while moving them away from each other. The formula is as follows: .

[0054] S207: Constructing cross-stage prototype loss: In continuous learning scenarios, in order to enable the model to retain the ability to retrieve old categories without accessing old data, this invention adopts a prototype alignment strategy to impose cross-stage constraints on the fused shared features.

[0055] In this embodiment, two prototypes are calculated for each visible category c at each learning stage: a sketch shared prototype and a sketch shared prototype. Sharing prototypes with images , defined as the mean of the shared features shared by all samples in that class. Let . The set of shared prototype sketches saved in each stage is The image sharing prototype set is ,in, For the first The number of categories in the stage.

[0056] For the current stage Given an input sketch, extract its shared feature vectors using both the old and current models, denoted as . and .

[0057] Similarly, for real image samples, the shared feature vectors are extracted using the old model and the current model, denoted as follows: and .

[0058] The present invention calculates the affinity distribution from the sketches extracted from the old model to the old prototypes in the real image as follows: ; in, For cosine similarity, This refers to the temperature parameter.

[0059] Similarly, the sketch features extracted by the current model The distribution relative to the old prototype of the same image is as follows: .

[0060] To preserve the decision boundary of old knowledge during continuous learning, the affinity relation matrix of the old model is used as the distillation objective, and the sketch image to be matched is minimized by KL divergence. The difference between the affinity distribution matrix of the new model and the affinity distribution matrix of the old model is used to construct the cross-stage prototype loss, which is expressed as: ; in, , .

[0061] This invention constructs a multi-layered alignment mechanism, using knowledge distillation across modal affinity distributions for alignment. This enables the model to balance new knowledge acquisition with the retention of old knowledge during continuous learning, significantly mitigating the catastrophic forgetting problem. Furthermore, by introducing a prototype-level representation method, it achieves indirect modeling of historical knowledge, avoiding the need to store raw data and improving the method's practicality in privacy-constrained scenarios.

[0062] S208: Constructing the total loss function: The total loss function is obtained by weighted summation of triplet loss, classification loss, text semantic-guided shared knowledge alignment loss, and cross-stage prototype alignment loss.

[0063] This embodiment uses the first encoding feature of the sketch after text semantic enhancement. As sketch anchor points, image features of the same category As positive samples, image features of different categories As negative samples, the distance between sketch image pairs of the same class is made smaller than the distance between pairs of different classes, and a triplet loss is constructed. The formula is: ; Where m is the interval parameter.

[0064] This embodiment uses the semantically enhanced first encoded feature Second coding features By inputting the weight-sharing classifier, the corresponding class prediction probability is obtained. and utilize real category labels Calculate cross-entropy loss and construct classification loss. , represented as: .

[0065] Total loss function The weighted sum of the triplet loss, classification loss, text semantic-guided shared knowledge alignment loss, and cross-stage prototype alignment loss is expressed as: ; in, The triplet loss is used to bring sketch-image pairs of the same category closer together and push apart pairs of different categories. Shared knowledge alignment loss guided by text semantics; This represents the cross-stage prototype alignment loss. and To balance the hyperparameters, they are set to 1.0 and 1.0 respectively in this invention. By minimizing this overall loss, the model can effectively retain the cross-modal retrieval capability of old categories while learning new categories.

[0066] This invention integrates a continuous learning mechanism into its overall framework, achieving refined modeling of cross-modal information through knowledge decoupling and alignment. First, based on a modality-sharing guidance mechanism, it dynamically determines which channels retain original style information and which adopt style-independent representations, depending on the specific circumstances of the current sample. This maximizes the retention of discriminative information while eliminating interference. Simultaneously, based on a modality-specific guidance mechanism, it independently models the unique expressive information of each modality. Through a cross-modal knowledge decoupling mechanism, shared and unique information are represented separately, reducing mutual interference. Furthermore, based on cross-modal knowledge alignment, constraints are applied to both types of knowledge during continuous learning, and category prototypes are introduced to uniformly model knowledge at different stages, enabling the model to maintain stable memory of old knowledge without relying on historical data. Through these mechanisms, a dynamic balance between new and old knowledge is achieved, improving retrieval performance in a continuous learning environment.

[0067] This invention utilizes modal shared guidance and modal-specific guidance mechanisms to explicitly decouple shared semantic information from unique expressive information in sketch and real image modalities, avoiding mutual interference between the two types of knowledge during continuous learning. It also introduces category text features as a semantic bridge to align the decoupled shared features across modalities, improving the discriminative power of retrieval. In each stage of continuous learning, only the shared prototypes of each type of sketch and image are saved. Using the old models and prototypes saved in the previous stage, the cross-modal affinity distribution between the current sample and the old prototype is calculated. A relation matrix distillation method is used to constrain the consistency of feature distributions between the new and old models, effectively mitigating catastrophic forgetting. This invention is trained sequentially on sequentially arriving sketch-image tasks, achieving the accumulation and maintenance of cross-modal retrieval capabilities in continuous scenarios. This method achieves excellent performance in average retrieval accuracy and anti-forgetting performance on several publicly available sketch retrieval continuous learning benchmarks.

[0068] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0069] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0070] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0071] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0072] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A cross-modal knowledge-decoupled sketch image retrieval method for continuous learning, characterized in that, include: The process involves acquiring the sketch image to be matched and its category text description for the current learning stage, as well as multiple real images. Feature encoding and dimensional reshaping are performed on each image to obtain sketch visual features, text features, and multiple sample visual features. This includes: obtaining the minimum anchor box containing the main subject based on black pixels in the sketch image; extracting the minimum anchor box containing the main subject, scaling it to a preset size, and then normalizing the mean and variance to obtain a preprocessed sketch; inputting the preprocessed sketch into a first image encoder to obtain first encoded features; reshaping the first encoded features into a 3D feature map of a preset size to obtain sketch visual features; scaling each real image to a preset size and then normalizing the mean and variance to obtain a preprocessed real image; inputting the preprocessed real image into a second image encoder to obtain second encoded features; reshaping the second encoded features into a 3D feature map of a preset size to obtain sample visual features; standardizing the category text description into text prompts based on a preset text template; and inputting the text prompts into a text encoder to obtain text features. Using the old model from the previous learning stage, obtain the old modality common features, old modality specific features, old shared feature vectors and old joint features corresponding to each visual feature; the visual features include sketch visual features and multiple sample visual features; By utilizing the new model in the current learning phase, we can obtain the new modality-common features, new modality-specific features, new shared feature vectors, and new joint features corresponding to each visual feature. Based on the InfoNCE loss function, the new shared feature vectors corresponding to each visual feature are made closer to the text features of their corresponding category text descriptions to construct a shared knowledge alignment loss. Obtain the old prototypes corresponding to each image category in the previous learning stage, and construct the cross-stage prototype loss based on the old joint features of sketch visual features and the cosine similarity of each old prototype, the new joint features of sketch visual features and the cosine similarity of each old prototype, and the KL divergence. The shared knowledge alignment loss, cross-stage prototype loss, triplet loss and classification loss are weighted and summed to obtain the total loss function. The new model is then trained to obtain the model trained in the current learning stage. The target real image is obtained by searching based on the sketch image to be matched and its category text description. The process involves using a model to obtain modal common features, modal-specific features, shared feature vectors, and joint features corresponding to each visual feature. This includes: normalizing the visual features to obtain normalized features; inputting the visual features and normalized features into a lightweight channel mask generation network to generate corresponding visual channel weight masks and normalized channel weight masks; using a channel-wise multiplication fusion method, weighting and fusing the visual features and normalized features using the visual channel weight masks and normalized channel weight masks to obtain the modal common features of the visual features; mapping the encoded features to modal-specific features based on a multilayer perceptron; flattening the modal common features to obtain shared feature vectors, and concatenating them with the modal-specific features along the channel dimension to obtain the joint features of the visual features. Among them, the visual features are either sketch visual features or sample visual features, and the corresponding coding features are either the first coding feature or the second coding feature. When the model is the old model from the previous learning stage, the old modality shared features, old modality-specific features, old shared feature vectors, and old joint features of the sketch visual features and the visual features of each sample are obtained. When the model is the new model in the current learning stage, the new modality shared features, new modality-specific features, new shared feature vectors, and new joint features of the sketch visual features and the visual features of each sample are obtained.

2. The cross-modal knowledge decoupling sketch image retrieval method for continuous learning as described in claim 1, characterized in that, After obtaining the first encoded feature and the text feature, the process also includes feature enhancement of the first encoded feature, including: The first encoded feature is linearly transformed and mapped to a query matrix, and the text features are mapped to a key matrix and a value matrix. Attention features are obtained based on the cross-attention mechanism. After performing a residual connection between the attention features and the first coding features, the enhanced first coding features are obtained through layer normalization.

3. The cross-modal knowledge decoupling sketch image retrieval method for continuous learning as described in claim 1, characterized in that, Visual features and normalized features are input into a lightweight channel mask generation network to generate corresponding visual channel weight masks and normalized channel weight masks, including: Learning stage Visual features in Visual channel weight mask , represented as: ； Learning stage Normalization features in Normalized channel weight mask , represented as: ； Among them, image categories , hour Visual channel weight mask representing the visual features of a sketch. hour A visual channel weight mask representing the visual features of a sample; hour Normalized channel weight mask representing the visual features of a sketch. hour Normalized channel weight mask representing the visual features of a sample; hour Indicate the visual features of the sketch, hour Indicates the visual features of the sample; hour Normalized features representing the visual characteristics of a sketch hour Normalized features representing the visual features of a sample; This represents the Sigmoid activation function. and Learnable parameters representing a lightweight channel mask generation network for visual feature input. Represents the ReLU nonlinear activation function. Indicates global average pooling. and The learnable parameters represent the lightweight channel mask generation network with normalized feature inputs.

4. The cross-modal knowledge decoupling sketch image retrieval method for continuous learning according to claim 3, characterized in that, Based on a channel-wise multiplication fusion method, visual features and normalized features are weighted and fused using visual channel weight masks and normalized channel weight masks to obtain the modal common features of the visual features, represented as: ； in, This indicates multiplication by channel.

5. The cross-modal knowledge decoupling sketch image retrieval method for continuous learning according to claim 1, characterized in that, Based on a multilayer perceptron, the encoded features are mapped to modality-specific features, represented as: ； in, Indicates the learning stage Mid-visual features Modal-specific characteristics, Representing visual features The corresponding multilayer perceptron, Indicates the learning stage Image categories The coding features, Indicates a random deactivation layer. Represents the ReLU activation function layer; , , , For learnable model parameters, , , This represents the hidden layer dimension of a multilayer perceptron. The dimension representing the encoded feature.

6. The cross-modal knowledge decoupling sketch image retrieval method for continuous learning according to claim 1, characterized in that, Based on the InfoNCE loss function, the new shared feature vector corresponding to each visual feature is made closer to the text features of its corresponding category text description to construct a shared knowledge alignment loss. , represented as: ； in, This represents the natural exponential function. Represents the cosine similarity function. Indicates the learning stage A new shared feature vector of visual features in sketches. Indicates the learning stage The middle category is The text features of the category text description This indicates the preset temperature parameter. Represents a set of categories. Indicates the learning stage A new shared feature vector of visual features of medium-sized samples.

7. The cross-modal knowledge decoupling sketch image retrieval method for continuous learning according to claim 1, characterized in that, Constructing cross-stage prototype loss includes: Based on the cosine similarity between the sketch visual features in the old joint features of the old model and each old prototype, the sketch image to be matched is calculated. With the The old affinity of the Gejiu prototype is represented as: ； Acquisition Learning Phase middle sketch image to be matched The old model affinity distribution matrix , represented as: ; in, Indicates based on learning stage Sketch images to be matched obtained from the old model The old characteristics of the union, Indicates the learning stage The first in An old prototype sample, Indicates temperature parameter; Indicates the learning stage The total number of old prototypes in the sample, Indicates the learning stage The first in One sample old prototype; Represents the row normalization function. Indicates transpose; Based on the new joint features of the sketch visual features in the new model and the cosine similarity of each old prototype, the sketch image to be matched is calculated. With the The New Affinity of Gejiu Prototype , represented as: ； Acquisition Learning Phase middle sketch image to be matched The affinity distribution matrix of the new model is expressed as: ; in, Indicates based on learning stage Sketch images to be matched obtained by the new model New joint features; Using the affinity matrix of the old model as the distillation objective, the sketch image to be matched is minimized by KL divergence. The difference between the affinity distribution matrix of the new model and the affinity distribution matrix of the old model is used to construct a cross-stage prototype loss. , represented as: 。 8. The cross-modal knowledge decoupling sketch image retrieval method for continuous learning according to claim 1, characterized in that, The total loss function is obtained by weighted summing of the shared knowledge alignment loss, cross-stage prototype loss, triplet loss, and classification loss, including: Using sketch visual features as anchor points, sample visual features of the same category as sketch visual features are used as positive samples, and sample visual features of different categories from sketch visual features are used as negative samples, to construct triplet loss; Based on the true category of the sketch image to be matched in the current learning stage and the category predicted by the new model, a classification loss is constructed; The total loss function is obtained by weighted summing of the shared knowledge alignment loss, cross-stage prototype loss, triplet loss, and classification loss.

Images