Visual positioning method, device, equipment and memory based on dual knowledge distillation

Abstract

The present application relates to a visual positioning method, apparatus, device and memory based on dual knowledge distillation. The method includes: taking the acquired original image and the corresponding language query as training samples; constructing a visual positioning model based on dual knowledge distillation; the model includes a student network, a semantic knowledge distillation module and a positioning knowledge distillation module; the semantic knowledge distillation module is used to encode the training samples into visual features and semantic features using a teacher network, and distill the visual features and semantic features into the student network; the positioning knowledge distillation module is used to learn positioning knowledge using a contrastive learning method; the multi-visual positioning model is trained according to the training samples and the total loss function, and the image to be tested and the corresponding language query are input into the student network of the trained visual positioning model to obtain a positioning bounding box. This method improves the cross-modal representation of the infrastructure and makes the correlation between the two modalities closer. The method can improve the accuracy of visual positioning.

Description

Technical Field

[0001] The present application relates to the field of visual positioning technology, and in particular to a visual positioning method, apparatus, device and memory based on dual knowledge distillation. Background Art

[0002] The visual localization task aims to locate the object referred to by a sentence in an image based on the referring language. Unlike general target detection tasks that locate a certain type of object in an image, the visual localization task requires the model to understand complex referring expressions and then distinguish the object that uniquely corresponds to the language from multiple similar objects in the image. This visual localization capability is a necessary condition for moving towards cross-media intelligence, and it is indispensable in many subsequent vision and language tasks, such as answering "the girl in green clothes" in the visual question answering task; in the visual language navigation task of "stopping next to the red table", the model needs to accurately locate the objects in the environment before it can carry out subsequent actions, that is, locate the "red table"; and in the autonomous driving task of "parking the car in parking space No. 1", it is necessary to locate the road position and then control the vehicle.

[0003] Knowledge distillation is a teacher-student training structure, where a pre-trained teacher model typically provides knowledge, and the student model acquires the teacher's knowledge through distillation training. It can transfer the knowledge of a complex teacher model to a simpler student model at the cost of slight performance loss. Model enhancement emphasizes leveraging other resources (such as unlabeled or cross-modal data) or knowledge distillation optimization strategies (such as mutual learning and self-learning) to improve the performance of a complex student model. For example, an unlabeled sample serves as input to both the teacher and student networks. A high-performance teacher network can usually predict the sample's label, which is then used to guide the training of a complex student network.

[0004] Self-distillation involves using a single network as both a teacher and a student model, allowing the network to improve its performance through knowledge distillation during self-learning. It is also a form of online knowledge distillation. Self-distillation can be broadly categorized into two types. The first involves using information from different samples for mutual distillation. The soft labels of other samples can prevent the network from making overconfident predictions and can even reduce intra-class distances by minimizing the distribution of predictions between different samples. Other work utilizes information from augmented samples, such as leveraging feature consistency across data under varying distortions, to promote robust learning within classes. Another type involves performing self-distillation between layers within a single network. The most common approach is to use features from a deeper network to guide the learning of a shallower network, where the features of the deeper network include the soft targets output by the network. In tasks involving sequential features, knowledge from previous frames is transferred to subsequent frames for learning. Learning within a single network's individual blocks can also be bidirectional, with each block collaborating and guiding each other's learning throughout the training process.

[0005] Existing visual localization methods can be divided into two categories: single-stage and two-stage approaches. Currently, many research works use Transformers as the encoder of the model to extract visual and language features. However, these methods still have two problems.

[0006] Problem 1: Most methods use independent pre-trained models for the visual and language encoders. Figure 1 As shown in (a), the TransVG model uses the parameters of DETR to initialize the visual encoder and the parameters of BERT to initialize the language encoder. Since the BERT and DETR models are pre-trained based on different tasks, the correlation between the features obtained after the semantically similar visual and language information is encoded by the encoder is poor. Recently, the visual language pre-training model CLIP has demonstrated its ability to connect visual and language modalities into a unified embedding space and has achieved remarkable results in multiple multimodal tasks. Intuitively, we can directly use the CLIP visual encoder and language encoder to replace the original encoder in the infrastructure, thereby ensuring that the features of the two modalities are closely related. As shown in Figure 1 As shown in (b), experimental results show that this simple replacement does not bring about performance improvement, but instead loses model accuracy. Therefore, we believe that the semantic knowledge contained in the CLIP model needs to be extracted in an indirect way.

[0007] Problem 2: In VG tasks, an image usually contains multiple objects or regions, requiring a powerful model to distinguish between the reference object (foreground) and other objects (background). However, existing methods often ignore this issue and easily confuse the two types of objects. This problem makes the model insensitive to significant foreground objects, resulting in ambiguity in position boundaries. Figure 1 As shown in (d) and (e), given the query "sky right of middle tree", TransVG fails to identify the foreground sky area and instead mistakenly focuses on the background building. These results show that processing challenging images containing multiple objects and with complex location information is very difficult. Summary of the Invention

[0008] Based on this, it is necessary to provide a visual positioning method, device, equipment and memory based on dual knowledge distillation to address the above technical problems.

[0009] A visual localization method based on dual knowledge distillation, the method comprising:

[0010] The obtained original images and corresponding language queries are used as training samples.

[0011] A visual localization model based on dual knowledge distillation is constructed; the visual localization model includes a student network, a semantic knowledge distillation module and a localization knowledge distillation module; the semantic knowledge distillation module is used to encode the training samples into visual features and semantic features using a teacher network, and distill the visual features and the semantic features from the teacher network to the student network; the student network is used to encode the training samples, and fuse the visual features and the semantic features according to the encoding results and the distillation, and make predictions based on the obtained fused features to obtain a predicted localization bounding box; the localization knowledge distillation module is used to generate high-quality positive and negative samples using a semantic localization-aware sampling mechanism based on the original image or feature vector of the predicted localization bounding box, and learn localization knowledge using a comparative learning method.

[0012] Construct a total loss function for the visual localization model.

[0013] The visual positioning model is trained according to the training samples and the total loss function to obtain a trained visual positioning model.

[0014] The image to be tested and the corresponding language query are input into the student network of the trained visual localization model to obtain the localization bounding box.

[0015] A visual positioning device based on dual knowledge distillation, comprising:

[0016] The training sample acquisition module is used to use the acquired original image and the corresponding language query as training samples.

[0017] A visual positioning model construction module based on dual knowledge distillation is used to construct a visual positioning model based on dual knowledge distillation; the visual positioning model includes a student network, a semantic knowledge distillation module and a positioning knowledge distillation module; the semantic knowledge distillation module is used to use a teacher network to encode the training samples into visual features and semantic features, and distill the visual features and the semantic features from the teacher network to the student network; the student network is used to encode the training samples, and fuse the visual features and the semantic features according to the encoding results and the distillation, and predict according to the obtained fusion features to obtain a predicted positioning bounding box; the positioning knowledge distillation module is used to generate high-quality positive and negative samples according to the original image or feature vector of the predicted positioning bounding box using a semantic positioning perception sampling mechanism, and learn positioning knowledge using a comparative learning method.

[0018] A visual positioning model training module based on dual knowledge distillation is used to construct a total loss function of the visual positioning model; the visual positioning model is trained according to the training samples and the total loss function to obtain a trained visual positioning model.

[0019] The visual localization module based on dual knowledge distillation is used to input the image to be tested and the corresponding language query into the student network of the trained visual localization model to obtain the localization bounding box.

[0020] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0021] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0022] The above-mentioned visual positioning method, device, equipment and memory based on dual knowledge distillation, the method includes: taking the acquired original image and the corresponding language query as training samples; constructing a visual positioning model based on dual knowledge distillation; the visual positioning model includes a student network, a semantic knowledge distillation module and a positioning knowledge distillation module; the semantic knowledge distillation module is used to use the teacher network to encode the training samples into visual features and semantic features, and distill the visual features and semantic features from the teacher network to the student network; the student network is used to encode the training samples, and fuse the encoding results with the distilled visual features and semantic features, and predict based on the obtained fusion features to obtain a predicted positioning bounding box; the positioning knowledge distillation module is used to generate high-quality positive and negative samples based on the predicted positioning bounding box using a semantic positioning perception sampling mechanism, and learn positioning knowledge using a contrastive learning method; constructing a total loss function of the visual positioning model; training the visual positioning model according to the training samples and the total loss function to obtain a trained visual positioning model; inputting the image to be tested and the corresponding language query into the student network of the trained visual positioning model to obtain a positioning bounding box. The model of this method significantly improves the cross-modal representation of the infrastructure and makes the correlation between the two modalities closer. This method can improve the accuracy of visual positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Introduced for this application problem, where (a) is the TransVG structure, (b) is the TransVG w / CLIP structure, (c) is a schematic diagram of the visual localization model structure based on dual knowledge distillation, (d) is the query "sky right of middletree", (e) is the visualization result of the TransVG attention score, and (f) is the visualization result of the visual localization model attention score based on dual knowledge distillation;

[0024] Figure 2 1 is a flow chart of a visual localization method based on dual knowledge distillation in one embodiment;

[0025] Figure 3 2 is a block diagram of a visual localization model based on dual knowledge distillation in one embodiment;

[0026] Figure 4 is an alignment module in another embodiment, wherein (a) is a feature-level alignment module, and (b) is a pixel-level alignment module;

[0027] Figure 5 1 is a structural block diagram of a visual positioning device based on dual knowledge distillation in one embodiment;

[0028] Figure 6 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0030] In one embodiment, Figure 2 As shown, a visual localization method based on dual knowledge distillation is provided, which includes the following steps:

[0031] Step 200: The obtained original image and the corresponding language query are used as training samples.

[0032] Step 202: Construct a visual localization model based on dual knowledge distillation; the visual localization model includes a student network, a semantic knowledge distillation module and a localization knowledge distillation module; the semantic knowledge distillation module is used to use the teacher network to encode the training samples into visual features and semantic features, and distill the visual features and semantic features from the teacher network to the student network; the student network is used to encode the training samples, and fuse the encoding results with the distilled visual features and semantic features, and make predictions based on the obtained fused features to obtain a predicted localization bounding box; the localization knowledge distillation module is used to generate high-quality positive and negative samples based on the original image or feature vector of the predicted localization bounding box using a semantic localization-aware sampling mechanism, and learn localization knowledge using a comparative learning method.

[0033] Specifically, the Dual Knowledge Distillation-based Visual Localization Model (DUET) introduces the knowledge distillation (KD) mechanism into the visual localization task. DUET includes two distillation methods: semantic knowledge and localization knowledge.

[0034] First, to have stronger semantic correlation between the feature embeddings of visual and linguistic modalities, semantic knowledge is extracted from CLIP by trying to match the global visual / linguistic features of the teacher network CLIP model and the target student network model, as Figure 1 This is a simple yet effective approach that can significantly improve the cross-modal representation of the base architecture and make the correlation between the two modalities closer.

[0035] Secondly, the semantic knowledge extracted from CLIP needs to be further refined, because CLIP is good at solving classification tasks, but has difficulty distinguishing foreground and background when solving visual localization tasks. To this end, contrastive learning is used as a self-distillation method to refine localization knowledge. Specifically, an alignment loss is established, which forces the features extracted from the bounding boxes produced by the model to be similar to positive samples and different from negative samples. However, when a large number of simple samples dominate the gradient, the performance of the model quickly enters a plateau. To address this issue, this method also designs a semantic location-aware (Semantic-Localization Aware, SLA) sampling mechanism to distinguish positive and negative samples from randomly cropped image regions. Figure 1 From the visualization results in (f), we can observe that DUET can help the baseline model generate unique location-aware representations for the visual localization task. The structure of the visual localization model based on dual knowledge distillation is shown in Figure 3 As shown in Figure 3, the DUET framework can be divided into two modules: semantic knowledge distillation module and positioning knowledge distillation module.

[0036] The student network may be, but is not limited to, a TransVG network or a FAOA network.

[0037] Step 204: Construct a total loss function of the visual localization model.

[0038] Step 206: Train the visual localization model according to the training samples and the total loss function to obtain a trained visual localization model.

[0039] Step 208: Input the image to be tested and the corresponding language query into the student network of the trained visual localization model to obtain a localization bounding box.

[0040] The above-mentioned visual localization method based on dual knowledge distillation includes: using the acquired original image and the corresponding language query as training samples; constructing a visual localization model based on dual knowledge distillation; the visual localization model includes a student network, a semantic knowledge distillation module, and a localization knowledge distillation module; the semantic knowledge distillation module is used to encode the training samples into visual features and semantic features using a teacher network, and distill the visual features and semantic features from the teacher network to the student network; the student network is used to encode the training samples, and fuse the encoding results with the distilled visual features and semantic features, and predict based on the obtained fused features to obtain a predicted localization bounding box; the localization knowledge distillation module is used to generate high-quality positive and negative samples based on the predicted localization bounding box using a semantic localization-aware sampling mechanism, and learn localization knowledge using a contrastive learning method; constructing a total loss function for the visual localization model; training the visual localization model based on the training samples and the total loss function to obtain a trained visual localization model; and inputting the image to be tested and the corresponding language query into the student network of the trained visual localization model to obtain a localization bounding box. The model of this method significantly improves the cross-modal representation of the basic architecture and makes the correlation between the two modalities closer. The method can improve the accuracy of visual localization.

[0041] In one embodiment, step 204 includes: constructing a loss function for the semantic knowledge distillation module, the positioning knowledge distillation module, and the student network; setting hyperparameters corresponding to the loss effects; and performing a weighted summation of the hyperparameters, the semantic knowledge distillation module, the positioning knowledge distillation module, and the loss function of the student network to obtain a total loss function for the visual positioning model:

[0042]

[0043] Among them, λ giou ,λ sem ,λ loc is the hyperparameter corresponding to the loss impact, L L1 and L giou They are the L1 loss and GIoU loss of the student network, L loc The loss function of the localization knowledge distillation module, L sem is the loss function of the semantic knowledge distillation module.

[0044] Specifically, GIoU loss plus L1 loss is used to solve the visual localization task. In order to perform knowledge distillation, we need to minimize two distillation losses, so the training objective of the visual localization model based on dual knowledge distillation is shown in formula (1).

[0045] In one embodiment, the loss function of the semantic knowledge distillation module is:

[0046] L sem =Lsem,v +L sem,l (2)

[0047]

[0048]

[0049] Among them, L sem is the loss function of the semantic knowledge distillation module, L sem,v and L sem,l denote the distillation losses of visual and language modalities respectively, is the overall visual feature of the student model, is the overall visual feature of the semantic knowledge distillation module, η(·) is the adaptive layer, w(·) is the feature whitening function, and β is the preset parameter; is the overall semantic feature of the student model, is the overall semantic feature of the semantic knowledge distillation module.

[0050] The loss function of the positioning knowledge distillation module is:

[0051]

[0052] in, They are the self-distillation loss using the pixel-level alignment module and the self-distillation loss using the feature-level alignment module, are image area and feature map area respectively, N pos and N neg are the number of positive and negative samples respectively.

[0053] In one embodiment, step 206 specifically includes the following steps:

[0054] Step 300: Input the training samples into the semantic knowledge distillation module for visual feature encoding and semantic feature encoding, perform whitening on the obtained encoding, and adaptively process the obtained whitened visual features and semantic features and distill them into the student network.

[0055] Step 302: Input the training sample into the student network to obtain the predicted positioning bounding box.

[0056] Step 304: Input the predicted positioning bounding box and the original image or feature vector in the corresponding area into the positioning knowledge distillation module to obtain positioning knowledge using a self-distillation method.

[0057] Step 306: Train the visual localization model based on the whitened visual features and semantic features, the distilled visual features and semantic features, the predicted localization bounding box, the localization knowledge obtained from the self-distillation, and the total loss function to obtain a trained visual localization model.

[0058] In one embodiment, the semantic knowledge distillation module includes a teacher network and a feature whitening module; the teacher network includes two large-scale pre-trained models CLIP; step 300 includes: inputting the training sample into the teacher network of the semantic knowledge distillation module and using the first large-scale pre-trained model CLIP to encode the visual features, and using the second large-scale pre-trained model CLIP to encode the semantic features to obtain visual features and semantic features; inputting the visual features and semantic features into the feature whitening module of the semantic knowledge distillation module and using a non-parametric layer normalization function without scaling or bias to perform whitening processing, and processing the whitened visual features and semantic features using an adaptive layer; distilling the adaptively processed visual features and semantic features into the student network.

[0059] Specifically, since the visual and language encoders of the student network use DETR and BERT pre-trained parameters respectively, there is a significant semantic gap between their embedding spaces. To bridge this gap, a large-scale pre-trained model CLIP is used, which is able to encode visual features and language features in the same semantic space. However, due to the differences between classification and regression tasks, directly using CLIP may not be optimal for reasoning tasks. Therefore, the first distillation method focuses on distilling semantic knowledge from the teacher network (CLIP) to the student network. It is worth noting that this efficient distillation method does not require additional pre-training and maintains the complexity of reasoning.

[0060] Specifically, using image I and language query L as input, the CLIP model encodes image I and language query L into visual features and language features. CLIP visual [CLS] token and CLIP text [EOS] token are used to represent the overall visual features V of the teacher network model respectively. t ∈R D and language features L t ∈R D At the same time, the output h of the student multimodal Transformer encoder s [REG]token, language features f l ' and visual features f v 'The output state is composed of. Select from the visual features f v 'All visual tokens and language features f l 'The first language token is used as the overall visual feature V of the student model s ∈R n×D and language features L s ∈R D, where n is the number of visual feature tokens. Before distillation, an adaptive layer and feature whitening are introduced to make the method more general.

[0061] (1) Adaptive layer. Since the overall features of the teacher and student models have different feature dimensions, we need to adjust the features to the same dimension to make them comparable. We have tested common methods such as average pooling and 1×1 convolution. The adaptive layer allows for different dimensions of output features between the teacher and student models, thus allowing our distillation method to be further generalized.

[0062] (2) Feature whitening. Adjusting the hyperparameters of various teacher network models can be challenging because different pre-trained models may have different feature magnitudes. To address this issue, a non-parametric layer normalization function without scaling or bias is used to whiten the output feature maps of the teacher model.

[0063] Denote the distillation of semantic knowledge as V s ∈R n×D and V t ∈R D , L s ∈R D and L t ∈R D Specifically, L sem is the sum of the distillation losses between the two modalities, and its expression is shown in Equation (2). For example, the loss for the visual modality can be calculated as the smoothed L1 loss between the overall features of the student and the teacher, as shown in Equation (3). Similarly, we can calculate the distillation loss L for the language modality sem,l , and the total distillation loss L is obtained sem .

[0064] In one embodiment, the student network is a TransVG network; the TransVG network includes a visual encoder, a language encoder, a multimodal fusion module, and a prediction layer; step 302 includes: inputting the original image in the training sample into the visual encoder of the student network to obtain visual features; inputting the language query corresponding to the original image in the training sample into the language encoder of the student network to obtain semantic features; inputting the semantic features, visual features, learnable tokens [REG], and the semantic features and visual features output by the distilled teacher network into the multimodal fusion module of the student network to obtain the fused features:

[0065] h s = Transformer([f REG ,φ(f l ),φ(f v )]) (6)

[0066] Among them, hs is the fusion feature, f l is the semantic feature, f v is the visual feature, φ(·) is the linear layer, f REG It is a learnable token [REG].

[0067] The fused features are input into the prediction layer of the student network to obtain the predicted localization bounding box.

[0068] Specifically, the dual knowledge distillation method proposed in this method can be applied to different models. Without loss of generality, this paper chooses the classic model TransVG as the student network. The student network consists of four main parts: visual encoder, language encoder, multimodal fusion module and prediction layer. Generally speaking, given an image I of size W×H and a language query L, the student network can output 4-point bounding box coordinates bbox in an end-to-end manner. pred =(x c ,y c ,ω,h), where x c ,y c are the coordinates of the center of the bounding box, and ω and h represent half of the width and height respectively.

[0069] (1) Visual encoder and language encoder. The visual encoder encodes the input image I through the convolutional network and then encodes the input image through the Transformer encoder layer to obtain the visual feature map f v Similarly, for each token in the language query L, we embed it as the sum of its word embedding and position embedding. Then, following the structure of the standard BERT model, we use a 6-layer Transformer to obtain the language feature f l .

[0070] (2) Multimodal fusion module. The multimodal fusion module includes a linear layer and a 6-layer Transformer encoder for each modality. The learnable token [REG], language feature f l and visual features f v They are concatenated and the visual and language features are fused using the Transformer encoder as shown in Equation (6).

[0071] (3) Prediction layer. To predict the coordinates of the localization box, a regression network with an MLP and activation function and a linear output layer is used. Given the output features of [REG], the prediction layer generates the 4D bounding box coordinates bbox pred =(x c ,y c ,ω,h).

[0072] In one embodiment, the positioning knowledge distillation module includes an alignment module and a semantic positioning-aware sampling mechanism; the alignment module includes a feature-level alignment module or a pixel-level alignment module; step 304 includes:

[0073] When the alignment module is a pixel-level alignment module: the predicted positioning bounding box and the original image in the corresponding area are input into the semantic positioning-aware sampling mechanism of the positioning knowledge distillation module to obtain multiple high-quality pixel-level positive and negative samples; the pixel-level alignment module is used to align all pixel-level positive and negative samples to obtain the relevant image area of each sample:

[0074]

[0075]

[0076] in, is the relevant image area, s is the grid sampling size, (x1, y1, x2, y2) are the coordinates of the upper left corner and lower right corner of the predicted positioning bounding box, W×H is the size adjustment; STN is the spatial transformer network, I is the pixel of the original image, and STN is the spatial transformer network;

[0077] When the alignment module is a feature-level alignment module: the predicted localization bounding box and the feature vector in the corresponding area are input into the semantic localization-aware sampling mechanism of the localization knowledge distillation module to obtain multiple high-quality feature-level positive and negative samples; the feature-level alignment module is used to perform pixel-level alignment on all feature-level positive and negative samples, and the relevant feature map area of each sample is obtained as follows:

[0078]

[0079]

[0080] in, is the feature map area, f v 'Visual features f v Visual features processed by the multimodal fusion module.

[0081] Specifically, for the visual localization task, simply giving the student model semantic knowledge is not enough to support its accurate output of localization boxes based on referential language. We believe that the student model also needs to learn how to distinguish between foreground and background. Therefore, localization knowledge is crucial for the success of the model in visual localization tasks, but is often ignored by existing methods. To solve this problem, we designed a region-level self-distillation method to learn localization knowledge. The regional features contained in the bounding box output from the student model should match the bounding box of the true label. In addition, the semantic localization-aware sampling mechanism is further utilized to generate more high-quality positive and negative samples, and the contrastive learning method is used to help the model learn localization knowledge.

[0082] (1) Feature-level alignment and pixel-level alignment

[0083] Given an image I and a query L, the student network can generate a bounding box bbox pred =(xc,yc,w,h). There are two different levels of methods to extract localization knowledge, namely feature-level alignment and pixel-level alignment. Figure 4 As shown in the figure, (a) is the pixel-level alignment module and (b) is the feature-level alignment module. Pixel-level alignment is to align the image pixels contained in the predicted positioning box with the image pixels contained in the ground-truth positioning box. Feature-level alignment aligns the feature vectors corresponding to the two regions. In order to shift the focus from the entire image or feature map to the region corresponding to the predicted bounding box bboxpred, the Spatial Transformer Network (STN) is used to obtain the pixel I or feature f while retaining the gradient information. v ' related regions on. In addition to its ability to map transformations, it is a differentiable image sampling method that allows the loss gradient to flow back to the input feature map and also to the sampling grid coordinates, and thus to the transformation parameters and the student network. First, since the pixel I or feature f v ' is different in size, the latter is resized to W×H to keep the same size as the former. Then, a transformation Tθ is defined as shown in Equation (8) based on a bounding box (x1, y1, x2, y2) containing the coordinates of the upper left corner and the lower right corner,

[0084] Based on this, relevant image regions can be extracted or feature map region As shown in formula (7) and formula (8).

[0085] (2) Semantic-localization Aware (SLA) sampling mechanism

[0086] We believe that introducing the above contrast will help absorb more positioning knowledge and benefit from more samples through contrastive learning. However, the random sampling method will reduce the performance of the model because the quality of the samples collected by this method is different. Therefore, the SLA sampling mechanism is proposed to generate high-quality positive and negative samples to help the model learn positioning knowledge. Specifically, first, the pixel I or feature f v 'Perform a fixed number of random cropping, denoted as N, to obtain a set of positioning boxes. By calculating the IoU value between all the positioning boxes in this set and the ground-truth boxes, the positioning boxes with IoU less than a certain threshold are regarded as negative samples. For samples with IoU greater than the threshold, calculate the language feature f l 'The cosine similarity between the regional features corresponding to the box. For samples whose IoU and cosine similarity are both greater than the threshold, we regard them as positive samples and number them as N pos , the rest of the samples are negative samples, numbered N neg The values of IoU and cosine similarity are set to 0.6 and normalized for these samples.

[0087] (3) Calculation of loss function

[0088] After SLA sampling, the predicted area R is calculated pred The self-distillation loss with other examples is:

[0089]

[0090] where R pred can be an image pixel in the image layer, i.e. get or feature maps in the feature layer, get N pos and N neg are the number of positive and negative samples respectively.

[0091] In addition, without using the SLA sampling mechanism and contrastive learning, we can perform naive localization knowledge distillation, i.e., aligning the predicted and true regions in two levels of L1 loss without any additional samples. and

[0092]

[0093] Where ||·||1 is the L1 norm, and are the generated ground-truth pixel area and feature area respectively.

[0094] It should be understood that although Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0095] In one embodiment, Figure 5 As shown, a visual positioning device based on dual knowledge distillation is provided, comprising: a training sample acquisition module, a visual positioning model construction module based on dual knowledge distillation, a visual positioning model training module based on dual knowledge distillation, and a visual positioning module based on dual knowledge distillation, wherein:

[0096] The training sample acquisition module is used to use the acquired original image and the corresponding language query as training samples.

[0097] A visual localization model construction module based on dual knowledge distillation is used to construct a visual localization model based on dual knowledge distillation; the visual localization model includes a student network, a semantic knowledge distillation module and a localization knowledge distillation module; the semantic knowledge distillation module is used to use a teacher network to encode training samples into visual features and semantic features, and distill the visual features and semantic features from the teacher network to the student network; the student network is used to encode the training samples, and fuse the encoding results with the distilled visual features and semantic features, and make predictions based on the obtained fused features to obtain a predicted localization bounding box; the localization knowledge distillation module is used to generate high-quality positive and negative samples based on the original image or feature vector of the predicted localization bounding box using a semantic localization-aware sampling mechanism, and learn localization knowledge using a contrastive learning method.

[0098] The visual localization model training module based on dual knowledge distillation is used to construct the total loss function of the visual localization model; the visual localization model is trained according to the training samples and the total loss function to obtain a trained visual localization model.

[0099] The visual localization module based on dual knowledge distillation is used to input the image to be tested and the corresponding language query into the student network of the trained visual localization model to obtain the localization bounding box.

[0100] In one embodiment, the visual localization model training module based on dual knowledge distillation is also used to construct the loss functions of the semantic knowledge distillation module, the localization knowledge distillation module, and the student network; set the hyperparameters corresponding to the loss impact; and according to the hyperparameters, the semantic knowledge distillation module, the localization knowledge distillation module, and the loss function of the student network, obtain the total loss function of the visual localization model as shown in formula (1).

[0101] In one embodiment, the loss function of the semantic knowledge distillation module is shown in equations (2) to (4). The loss function of the positioning knowledge distillation module is shown in equation (2).

[0102] In one embodiment, the visual positioning model training module based on dual knowledge distillation is also used to input the training samples into the semantic knowledge distillation module for visual feature encoding and semantic feature encoding, whiten the obtained encoding, and adaptively process the obtained whitened visual features and semantic features and distill them into the student network; input the training samples into the student network to obtain a predicted positioning bounding box; input the predicted positioning bounding box and the original image or feature vector in the corresponding area into the positioning knowledge distillation module to obtain positioning knowledge using a self-distillation method; train the visual positioning model based on the whitened visual features and semantic features, the distilled visual features and semantic features, the predicted positioning bounding box, the positioning knowledge obtained by self-distillation, and the total loss function to obtain a trained visual positioning model.

[0103] In one embodiment, the semantic knowledge distillation module includes a teacher network and a feature whitening module; the teacher network includes two large-scale pre-trained models CLIP; the visual positioning model training module based on dual knowledge distillation is further used to input the training samples into the teacher network of the semantic knowledge distillation module, use the first large-scale pre-trained model CLIP for visual feature encoding, and use the second large-scale pre-trained model CLIP for semantic feature encoding to obtain visual features and semantic features; input the visual features and semantic features into the feature whitening module of the semantic knowledge distillation module, use a non-parametric layer normalization function without scaling or bias to perform whitening processing, and use an adaptive layer to process the whitened visual features and semantic features; distill the adaptively processed visual features and semantic features into the student network.

[0104] In one embodiment, the student network is a TransVG network; the TransVG network includes a visual encoder, a language encoder, a multimodal fusion module, and a prediction layer; the visual localization model training module based on dual knowledge distillation is further used to input the original image in the training sample into the visual encoder of the student network to obtain visual features; the language query corresponding to the original image in the training sample is input into the language encoder of the student network to obtain semantic features; the semantic features, visual features, learnable token [REG], and the semantic features and visual features output by the distilled teacher network are input into the multimodal fusion module of the student network to obtain fusion features as shown in formula (6); the fusion features are input into the prediction layer of the student network to obtain a predicted localization bounding box.

[0105] In one embodiment, the positioning knowledge distillation module includes an alignment module and a semantic positioning perception sampling mechanism; the alignment module includes a feature-level alignment module or a pixel-level alignment module; the visual positioning model training module based on dual knowledge distillation is further used for: when the alignment module is a pixel-level alignment module: inputting the predicted positioning bounding box and the original image in the corresponding area into the semantic positioning perception sampling mechanism of the positioning knowledge distillation module to obtain multiple high-quality pixel-level positive samples and negative samples; using the pixel-level alignment module to perform pixel-level alignment on all pixel-level positive samples and negative samples, and obtaining the relevant image area of each sample as shown in formulas (7) and (8); when the alignment module is a feature-level alignment module: inputting the predicted positioning bounding box and the feature vector in the corresponding area into the semantic positioning perception sampling mechanism of the positioning knowledge distillation module to obtain multiple high-quality feature-level positive samples and negative samples; using the feature-level alignment module to perform pixel-level alignment on all feature-level positive samples and negative samples, and obtaining the relevant feature mapping area of each sample as shown in formulas (9) and (10).

[0106] For the specific definition of the visual positioning device based on dual knowledge distillation, please refer to the definition of the visual positioning method based on dual knowledge distillation above, which will not be repeated here. The various modules in the above-mentioned visual positioning device based on dual knowledge distillation can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0107] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a visual positioning method based on dual knowledge distillation is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0108] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0109] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiment when executing the computer program.

[0110] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiment are implemented.

[0111] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0112] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0113] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A visual positioning method based on dual knowledge distillation, characterized in that: The method comprises: The obtained original image and the corresponding language query are used as training samples; Construct a visual localization model based on dual knowledge distillation; the visual localization model includes a student network, a semantic knowledge distillation module, and a localization knowledge distillation module; the semantic knowledge distillation module is used to encode the training samples into visual features and semantic features using a teacher network, and distill the visual features and the semantic features from the teacher network to the student network; the student network is used to encode the training samples, and fuse the visual features and the semantic features according to the encoding results and the distillation, and predict based on the obtained fused features to obtain a predicted localization bounding box; the localization knowledge distillation module is used to generate high-quality positive and negative samples based on the original image or feature vector of the predicted localization bounding box using a semantic localization-aware sampling mechanism, and learn localization knowledge using a comparative learning method; Constructing a total loss function of the visual localization model; Training the visual positioning model according to the training samples and the total loss function to obtain a trained visual positioning model; The image to be tested and the corresponding language query are input into the student network of the trained visual localization model to obtain the localization bounding box.

2. The method according to claim 1, characterized in that Constructing the total loss function of the visual localization model includes: Constructing the semantic knowledge distillation module, the positioning knowledge distillation module, and the loss function of the student network; Set the hyperparameters corresponding to the loss impact; According to the hyperparameters, the semantic knowledge distillation module, the positioning knowledge distillation module and the loss function of the student network, the total loss function of the visual positioning model is obtained as follows: Among them, λ giou ,λ sem ,λ loc is the hyperparameter corresponding to the loss impact, and L giou They are the L1 loss and GIoU loss of the student network, L loc To locate the loss function of the knowledge distillation module, L sem is the loss function of the semantic knowledge distillation module.

3. The method according to claim 2, characterized in that The loss function of the semantic knowledge distillation module is: L sem =L sem,v +L sem,l Among them, L sem is the loss function of the semantic knowledge distillation module, L sem,v and L sem,l denote the distillation losses of visual and language modalities respectively, is the overall visual feature of the student model, is the overall visual feature of the semantic knowledge distillation module, η(·) is the adaptive layer, w(·) is the feature whitening function, and β is the preset parameter; is the overall semantic feature of the student model, is the overall semantic feature of the semantic knowledge distillation module; The loss function of the positioning knowledge distillation module is: in, They are the self-distillation loss using the pixel-level alignment module and the self-distillation loss using the feature-level alignment module, are image area and feature map area respectively, N pos and N neg are the number of positive and negative samples respectively.

4. The method according to claim 1, wherein The visual positioning model is trained according to the training sample and the total loss function to obtain a trained visual positioning model, including: Inputting the training samples into the semantic knowledge distillation module for visual feature encoding and semantic feature encoding, performing whitening processing on the obtained encoding, and adaptively processing the obtained visual features and semantic features after whitening and distilling them into the student network; Inputting the training sample into the student network to obtain a predicted positioning bounding box; Inputting the predicted positioning bounding box and the original image or feature vector in the corresponding area into the positioning knowledge distillation module to obtain positioning knowledge using a self-distillation method; The visual positioning model is trained based on the whitened visual features and semantic features, the distilled visual features and semantic features, the predicted positioning bounding box, the positioning knowledge obtained from the distillation, and the total loss function to obtain a trained visual positioning model.

5. The method according to claim 4, characterized in that The semantic knowledge distillation module includes a teacher network and a feature whitening module; the teacher network includes two large-scale pre-training models CLIP; Inputting the training sample into the semantic knowledge distillation module for visual feature encoding and semantic feature encoding, performing whitening processing on the obtained encoding, and adaptively processing the obtained visual features and semantic features after whitening processing and distilling them into the student network, including: Inputting the training samples into the teacher network of the semantic knowledge distillation module, using the first large-scale pre-trained model CLIP to encode visual features, and using the second large-scale pre-trained model CLIP to encode semantic features, to obtain visual features and semantic features; Inputting the visual features and the semantic features into the feature whitening module of the semantic knowledge distillation module, performing whitening processing using a non-parametric layer normalization function without scaling or bias, and processing the whitened visual features and semantic features using an adaptive layer; The adaptively processed visual features and the semantic features are distilled into the student network.

6. The method according to claim 4, characterized in that The student network is a TransVG network; the TransVG network includes a visual encoder, a language encoder, a multimodal fusion module and a prediction layer; Input the training sample into the student network to obtain a predicted positioning bounding box, including: Inputting the original image in the training sample into the visual encoder of the student network to obtain visual features; Inputting the language query corresponding to the original image in the training sample into the language encoder of the student network to obtain semantic features; The semantic features, the visual features, the learnable token [REG], and the semantic features and visual features output by the distilled teacher network are input into the multimodal fusion module of the student network to obtain the fused features: h s =Transformer([f REG ,φ(f l ),φ(f v )]) Among them, h s is the fusion feature, f l is the semantic feature, f v is the visual feature, φ(·) is the linear layer, f REG is the learnable token [REG]; The fused features are input into the prediction layer of the student network to obtain a predicted positioning bounding box.

7. The method according to claim 4, characterized in that The positioning knowledge distillation module includes an alignment module and a semantic positioning perception sampling mechanism; the alignment module includes a feature level alignment module or a pixel level alignment module; Inputting the predicted positioning bounding box and the original image or feature vector in the corresponding area into the positioning knowledge distillation module and adopting the self-distillation method to obtain positioning knowledge, including: When the alignment module is the pixel-level alignment module: the predicted positioning bounding box and the original image in the corresponding area are input into the semantic positioning perception sampling mechanism of the positioning knowledge distillation module to obtain multiple high-quality pixel-level positive samples and negative samples; all pixel-level positive samples and negative samples are aligned at the pixel level using the pixel-level alignment module to obtain the relevant image area of each sample: in, is the relevant image area, s is the grid sampling size, (x1, y1, x2, y2) are the coordinates of the upper left corner and lower right corner of the predicted positioning bounding box, W×H is the size adjustment; STN is the spatial transformer network, I is the pixel of the original image, and STN is the spatial transformer network; When the alignment module is the feature-level alignment module: the predicted positioning bounding box and the feature vectors in the corresponding area are input into the semantic positioning-aware sampling mechanism of the positioning knowledge distillation module to obtain multiple high-quality feature-level positive samples and negative samples; the feature-level alignment module is used to perform pixel-level alignment on all feature-level positive samples and negative samples, and the relevant feature mapping area of each sample is obtained as follows: in, is the feature map area, f′ v is the visual feature f v Visual features processed by the multimodal fusion module.

8. A visual positioning device based on dual knowledge distillation, characterized in that: The device comprises: A training sample acquisition module is used to use the acquired original image and the corresponding language query as training samples; A visual localization model construction module based on dual knowledge distillation is used to construct a visual localization model based on dual knowledge distillation; the visual localization model includes a student network, a semantic knowledge distillation module and a localization knowledge distillation module; the semantic knowledge distillation module is used to encode the training samples into visual features and semantic features using a teacher network, and distill the visual features and the semantic features from the teacher network to the student network; the student network is used to encode the training samples, and fuse the visual features and the semantic features according to the encoding results and the distillation, and make predictions based on the obtained fused features to obtain a predicted localization bounding box; the localization knowledge distillation module is used to generate high-quality positive and negative samples based on the original image or feature vector of the predicted localization bounding box using a semantic localization-aware sampling mechanism, and learn localization knowledge using a contrastive learning method; A visual localization model training module based on dual knowledge distillation is used to construct a total loss function of the visual localization model; the visual localization model is trained according to the training samples and the total loss function to obtain a trained visual localization model; The visual localization module based on dual knowledge distillation is used to input the image to be tested and the corresponding language query into the student network of the trained visual localization model to obtain the localization bounding box.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer readable memory having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images