Single-stage weak supervision target positioning method capable of learning vector features for comparative learning, electronic equipment and storage medium
By using the method of learningable vector features for comparison learning in weakly supervised visual positioning, the problem of fragmentation of Anchor feature representation and excessive receptive field is solved, and more efficient and accurate visual-language alignment is achieved.
Patent Information
- Application Number
- CN202411826002.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-12
- Publication Date
- 2025-04-15
AI Technical Summary
The existing weakly supervised visual positioning method relies on Anchor features, and there are problems such as fragmentation of object representation and excessive receptive field, resulting in inaccurate visual-language alignment.
A single-stage weak-supervised target positioning method for comparative learning of learnable vector features is used, image features are extracted through Transformer decoder, cross-modal feature mapping is performed with text encoder, and feature selection is optimized through comparison learning.
The trade-off between the accuracy and inference speed of visual positioning tasks is improved, achieving better overall performance.
Smart Images
Figure CN119832356A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of weakly supervised visual positioning, and relates to realizing target positioning by utilizing low-cost annotation information, and in particular refers to a single-stage weakly supervised target positioning method, electronic device and storage medium that can perform contrastive learning on learnable vector features. Background Art
[0002] Visual localization is an important task in the multimedia field, which aims to locate the target object in the image based on the natural language description. Compared with traditional detection tasks, visual localization not only needs to capture detailed object information, but also needs to achieve accurate visual-language alignment. To this end, existing methods usually require expensive annotations to learn the accurate alignment between natural language descriptions and objects, which will cost huge manpower, material and financial resources for actual deployment.
[0003] To overcome this limitation, researchers have devoted themselves to weakly supervised learning for visual localization. Among them, early methods often rely on two-stage target detectors to frame weakly supervised visual localization as a Region-Text matching problem. However, these methods are often limited in their inference speed due to the expensive processing overhead of the two-stage detector. Therefore, recent research has focused on efficient one-stage models, which first extract anchor features from a pre-trained one-stage detector and then perform anchor-text matching to select target anchors. Finally, a detection head is used to decode the anchors to bounding boxes. This one-stage paradigm does not rely on complex processing such as RoI pooling and exhibits higher efficiency than the two-stage approach.
[0004] However, the anchor-based weak supervision framework is still limited by its object representation. Anchors are grid features extracted by pre-trained detectors, such as YOLOv3. Ideally, these anchors should represent the corresponding object information so that Anchor-Text matching can select the target anchor for positioning. However, compared with Region features, Anchors are fragmented and difficult to accurately describe object information. In addition, the large receptive field of Anchor often makes it receive more noisy object information, which inevitably hinders accurate visual-language alignment. Summary of the invention
[0005] The purpose of the present invention is to provide a single-stage weakly supervised target positioning method, electronic device and storage medium for comparative learning of learnable vector features, so as to solve the problems existing in the prior art and improve the performance of weakly supervised visual positioning tasks.
[0006] In order to achieve the above object, the solution of the present invention is: A single-stage weakly supervised target localization method for contrastive learning of learnable vector features comprises the following steps: Step 1: Obtain the learnable vector features that have interacted with the input image information through the Transformer decoder. The calculation formula is: ; ; ; in, represents the number of layers of the Transformer decoder, , and They represent the Transformer decoder layer The output of the self-attention layer, the cross-attention layer, and the feed-forward network layer; Representation layer normalization; represents random dropout; represents the normalized exponential function; Indicates The learnable vector features of the layer, represents the real number space, represents the number of learnable vectors per layer, Represents the feature dimension of each layer's learnable vector, defined as , Indicates Layer learnable vectors, The subscript representing the learnable vector of each layer Range is ; represents the matrix transpose; The height and width of the spatial resolution of the layer image feature are respectively and , the transformed image feature is and ; represents a multilayer perceptron; the positional encoding is denoted by ,have: ; ; ; ; Among them, all All represent linear transformations; the total number of layers is The learnable vector feature after the final processing of the Transformer decoder is recorded as ; Step 2: Filter according to the confidence score of each learnable vector feature and keep only The learnable vector features with the highest confidence scores are selected as candidate features, and the calculation formula is: ; in, represents the candidate learnable vector features selected based on the confidence score; Indicates that the learnable vector features are sorted in descending order according to the confidence score; Indicates the subscript range of the selected learnable vector features for ; Indicates learnable vector features; Step 3: Perform cross-modal feature mapping to map the candidate learnable vector features and the text features encoded by the text encoder to the same semantic space. The calculation formula is: ; ; in, Represents the learnable vector features after mapping; Represents the text features after mapping; and Represent the projection matrices of learnable vector features and text features respectively; and Represent the bias items of learnable vector features and text features respectively; Represents text features; Step 4: Estimate the quality of the learnable vector features. The calculation formula is: ; ; in, and They represent the uniqueness and distinction difficulty of the learnable vector features in the negative sample feature space of contrastive learning respectively; Indicates the iteration round; Indicates min-max normalization; Indicates the number of negative samples that have been selected into the contrastive learning feature space so far; The subscript representing the negative samples that have been selected into the contrastive learning feature space Range is ; Indicates the calculation of cosine similarity; Represents the negative samples of learnable feature vectors that have been previously selected into the contrastive learning feature space; Indicates the calculation of similarity; The final negative sample quality score is defined as: ; in, Represents the quality estimate of the learnable vector feature; represents a natural constant; represents the magnification factor; Step 5: Select from each image-text pair The learnable vector features are iteratively selected to make full use of the weak supervision information of the image-text pair. The calculation formula is: ; in, and Respectively represent , The set of negative sample features represented by the round; Indicates The learnable vector feature with the largest quality estimation value among the learnable vector features selected in rounds; Represents the union of sets; Step 6: Comparative learning based on learnable vector features: Select the learnable vector features with the highest similarity score from the matching image-text pairs as positive samples, and select several learnable vector features from the unmatched image-text pairs as negative samples. The optimization goal is to maximize the similarity between the text features and the learnable vector features of the positive samples, while minimizing the similarity between the text features and the learnable vector features of the negative samples. The calculation formula is: ; in, Represents the contrastive learning loss function used in the training model; Represents exponential operation; represents a training sample selected from a batch of data, is a positive sample; Indicates Text features of the batch; represents the temperature parameter; represents the number of negative samples selected from each mismatched image-text pair, Indicates the number of times the learnable vector features are iteratively selected Range is ; represents the batch size, Indicates the subscript range of the selected learnable vector features for ; represents the indicator function, when and When its value is 0.
[0007] Preferably, in step 4, after calculating the quality scores of the negative samples, they are sorted in descending order to select an appropriate number of negative samples for comparative learning.
[0008] Preferably, in step 1, the learnable vector features output by the Transformer decoder in the pre-trained DETR based on the Resnet-101 backbone network are used for comparative learning.
[0009] Preferably, in step 2, 20 learnable vector features with the highest confidence scores are selected as candidate learnable vector features for comparative learning; in step 3, the text encoder adopts bidirectional GRU, and the dimensions of the candidate learnable vector features and the text mapping features are set to 512 or 1024; in step 6, AdamW is used as the optimizer, the batch size is set to 32, the learning rate is set to 1e-4, and the training rounds are set to 25.
[0010] An electronic device comprises a processor, a memory and an application program; the application program is stored in the memory and is configured so that the processor executes the single-stage weakly supervised target localization method for comparative learning of learnable vector features.
[0011] A computer-readable storage medium, characterized in that the storage medium stores a computer program; when the computer program is executed in the computer, the computer executes the single-stage weakly supervised target localization method for contrastive learning of learnable vector features.
[0012] After adopting the above technical solution, the present invention has the following technical effects: It has a better trade-off between accuracy and inference speed, achieving better overall performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Figure 1 It is a model overview diagram of a specific embodiment of the present invention; Figure 2 Diagram of iterative selection of updates for learnable vectors; Figure 3 A comparison chart of a two-stage framework for matching based on regional features, a single-stage framework for matching based on anchor features, and a single-stage framework for matching based on learnable vector features. DETAILED DESCRIPTION
[0014] In order to further explain the technical solution of the present invention, the present invention is described in detail below through specific embodiments.
[0015] refer to Figure 1-2As shown, the present invention discloses a single-stage weakly supervised target localization method for contrastive learning of learnable vector features, comprising the following steps: 1. Model Implementation Process 1.1 Design of a single-stage contrastive learning framework based on learnable vector features Step 1: Obtain the learnable vector features that have interacted with the input image information through the Transformer decoder. The calculation formula is: ; ; ; in, represents the number of layers of the Transformer decoder, , and They represent the Transformer decoder layer The output of the self-attention layer, the cross-attention layer, and the feed-forward network layer; Representation layer normalization; represents random dropout; represents the normalized exponential function; Indicates The learnable vector features of the layer, represents the real number space, represents the number of learnable vectors per layer, Represents the feature dimension of each layer's learnable vector, defined as , Indicates Layer learnable vectors, The subscript representing the learnable vector of each layer Range is ; represents the matrix transpose; The height and width of the spatial resolution of the layer image feature are respectively and , the transformed image feature is and ; represents a multilayer perceptron; the positional encoding is denoted by ,have: ; ; ; ; Among them, all All represent linear transformations; the total number of layers is The learnable vector feature after the final processing of the Transformer decoder is recorded as ; Step 2: Filter according to the confidence score of each learnable vector feature and keep only The learnable vector features with the highest confidence scores are selected as candidate features, and the calculation formula is: ; in, represents the candidate learnable vector features selected based on the confidence score; Indicates that the learnable vector features are sorted in descending order according to the confidence score; Indicates the subscript range of the selected learnable vector features for ; Indicates learnable vector features; Step 3: Perform cross-modal feature mapping to map the candidate learnable vector features and the text features encoded by the text encoder to the same semantic space. The calculation formula is: ; ; in, Represents the learnable vector features after mapping; Represents the text features after mapping; and Represent the projection matrices of learnable vector features and text features respectively; and Represent the bias items of learnable vector features and text features respectively; Represents text features; Step 4: Estimate the quality of the learnable vector features. The calculation formula is: ; ; in, and They represent the uniqueness and distinction difficulty of the learnable vector features in the negative sample feature space of contrastive learning respectively; Indicates the iteration round; Indicates min-max normalization; Indicates the number of negative samples that have been selected into the contrastive learning feature space so far; The subscript representing the negative samples that have been selected into the contrastive learning feature space Range is ; Indicates the calculation of cosine similarity; Represents the negative samples of learnable feature vectors that have been previously selected into the contrastive learning feature space; Indicates the calculation of similarity; The final negative sample quality score is defined as: ; in, Represents the quality estimate of the learnable vector feature; represents a natural constant; represents the magnification factor; Step 5: Select from each image-text pair The learnable vector features are iteratively selected to make full use of the weak supervision information of the image-text pair. The calculation formula is: ; in, and Respectively represent , The set of negative sample features represented by the round; Indicates The learnable vector feature with the largest quality estimation value among the learnable vector features selected in rounds; Represents the union of sets; Step 6: Comparative learning based on learnable vector features: Select the learnable vector features with the highest similarity score from the matching image-text pairs as positive samples, and select several learnable vector features from the unmatched image-text pairs as negative samples. The optimization goal is to maximize the similarity between the text features and the learnable vector features of the positive samples, while minimizing the similarity between the text features and the learnable vector features of the negative samples. The calculation formula is: ; in, Represents the contrastive learning loss function used in the training model; Represents exponential operation; represents a training sample selected from a batch of data, is a positive sample; Indicates Text features of the batch; represents the temperature parameter; represents the number of negative samples selected from each mismatched image-text pair, Indicates the number of times the learnable vector features are iteratively selected Range is ; represents the batch size, Indicates the subscript range of the selected learnable vector features for ; represents the indicator function, when and When its value is 0.
[0016] 1.2 Deployment of a single-stage weakly supervised visual localization model based on contrastive learning of learnable vector features See also Figure 1-2 , the reasoning process of the trained single-stage weakly supervised visual localization model based on learnable vector feature contrast learning is as follows: ; ; .
[0017] The most similar learnable vector features can be obtained by inner product matching of text features in the same feature space and learnable vector features containing visual information, and then the detection results can be obtained through the subsequent decoding module. The calculation formula is: ; ; in, Represents the learnable vector feature that best matches the input text feature; Represents a set of learnable vector features; Represents input text; represents the input image; Represents learnable vector features; Indicates the best matching detection result; Represents the subsequent decoding module.
[0018] 2. Model Training Process See also Figure 3The present invention uses RefCOCO, RefCOCO+ and RefCOCOg datasets to perform visual localization tasks, including directional target detection and segmentation. The RefCOCO dataset includes 142,210 natural language descriptions and 50,000 targets, mainly including information based on absolute spatial position; RefCOCO+ includes 141,564 natural language descriptions and 49,856 targets, mainly including information based on relative spatial position and object appearance; RefCOCOg includes 104,560 natural language descriptions and 54,822 targets. Compared with RefCOCO and RefCOCO+, RefCOCOg has longer and more complex object descriptions. In the above task, the present invention uses the pre-trained DETR based on the Resnet-101 backbone network as the visual encoding module and the bidirectional GRU as the text encoder; the feature dimension of contrastive learning is set to 512 or 1024, the number of candidate learnable feature vectors is set to 20, and the temperature value of the softmax function is set to 10; at the same time, the present invention uses AdamW as the optimizer, the batch size is set to 32, the learning rate is set to 1e-4, and the model is trained for 25 rounds.
[0019] Through the above scheme, see Figure 3 ,Compared with existing methods, the present method has a better trade-off between accuracy and ,inference speed, and can achieve better overall performance.
[0020] Specific embodiments of the present invention are shown below.
[0021] In step 4, after calculating the quality scores of the negative samples, they are sorted in descending order to select an appropriate number of negative samples for comparative learning.
[0022] In the step 1, the learnable vector features output by the Transformer decoder in the pre-trained DETR based on the Resnet-101 backbone network are used for comparative learning.
[0023] In the step 2, 20 learnable vector features with the highest confidence scores are selected as candidate learnable vector features for comparative learning; in the step 3, the text encoder adopts a bidirectional GRU, and the dimensions of the candidate learnable vector features and the text mapping features are set to 512 or 1024; in the step 6, AdamW is used as the optimizer, the batch size is set to 32, the learning rate is set to 1e-4, and the training rounds are set to 25.
[0024] The present invention also discloses an electronic device, comprising a processor, a memory and an application program; the application program is stored in the memory and is configured so that the processor executes the above method.
[0025] The present invention also discloses a computer-readable storage medium, which stores a computer program; when the computer program is executed in a computer, the computer executes the above method. The storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
[0026] The above embodiments and drawings do not limit the product form and style of the present invention. Any appropriate changes or modifications made by ordinary technicians in the relevant technical field should be deemed to be within the patent scope of the present invention.
Claims
1. A single-stage weakly supervised target localization method that can learn vector features for contrastive learning, characterized in that The following steps are involved: Step 1: Obtain the learnable vector features that have interacted with the input image information through the Transformer decoder. The calculation formula is: ； ； ； in, represents the number of layers of the Transformer decoder, , and They represent the Transformer decoder layer The output of the self-attention layer, the cross-attention layer, and the feed-forward network layer; Representation layer normalization; represents random dropout; represents the normalized exponential function; Indicates The learnable vector features of the layer, represents the real number space, represents the number of learnable vectors per layer, Represents the feature dimension of each layer's learnable vector, defined as , Indicates Layer learnable vectors, The subscript representing the learnable vector of each layer Range is ; represents the matrix transpose; The height and width of the spatial resolution of the layer image feature are respectively and , the transformed image feature is and ; represents a multilayer perceptron; the positional encoding is denoted by ,have: ； ； ； ； Among them, all All represent linear transformations; the total number of layers is The learnable vector feature after the final processing of the Transformer decoder is recorded as ; Step 2: Filter according to the confidence score of each learnable vector feature and keep only The learnable vector features with the highest confidence scores are selected as candidate features, and the calculation formula is: ； in, represents the candidate learnable vector features selected based on the confidence score; Indicates that the learnable vector features are sorted in descending order according to the confidence score; Indicates the subscript range of the selected learnable vector features for ; Indicates learnable vector features; Step 3: Perform cross-modal feature mapping to map the candidate learnable vector features and the text features encoded by the text encoder to the same semantic space. The calculation formula is: ； ； in, Represents the learnable vector features after mapping; Represents the text features after mapping; and Represent the projection matrices of learnable vector features and text features respectively; and Represent the bias items of learnable vector features and text features respectively; Represents text features; Step 4: Estimate the quality of the learnable vector features. The calculation formula is: ； ； in, and They represent the uniqueness and distinction difficulty of the learnable vector features in the negative sample feature space of contrastive learning respectively; Indicates the iteration round; Indicates min-max normalization; Indicates the number of negative samples that have been selected into the contrastive learning feature space so far; The subscript representing the negative samples that have been selected into the contrastive learning feature space Range is ; Indicates the calculation of cosine similarity; Represents the negative samples of learnable feature vectors that have been previously selected into the contrastive learning feature space; Indicates the calculation of similarity; The final negative sample quality score is defined as: ； in, Represents the quality estimate of the learnable vector feature; represents a natural constant; represents the magnification factor; Step 5: Select from each image-text pair The learnable vector features are iteratively selected to make full use of the weak supervision information of the image-text pair. The calculation formula is: ； in, and Respectively represent , The set of negative sample features represented by the round; Indicates The learnable vector feature with the largest quality estimation value among the learnable vector features selected in rounds; Represents the union of sets; Step 6: Comparative learning based on learnable vector features: Select the learnable vector features with the highest similarity score from the matching image-text pairs as positive samples, and select several learnable vector features from the unmatched image-text pairs as negative samples. The optimization goal is to maximize the similarity between the text features and the learnable vector features of the positive samples, while minimizing the similarity between the text features and the learnable vector features of the negative samples. The calculation formula is: ； in, Represents the contrastive learning loss function used in the training model; Represents exponential operation; represents a training sample selected from a batch of data, is a positive sample; Indicates Text features of the batch; represents the temperature parameter; represents the number of negative samples selected from each mismatched image-text pair, Indicates the number of times the learnable vector features are iteratively selected Range is ; represents the batch size, Indicates the subscript range of the selected learnable vector features for ; represents the indicator function, when and When its value is 0.
2. The single-stage weakly supervised target localization method for contrastive learning of learnable vector features according to claim 1, characterized in that: In step 4, after calculating the quality scores of the negative samples, they are sorted in descending order to select an appropriate number of negative samples for comparative learning.
3. The single-stage weakly supervised target localization method for contrastive learning of learnable vector features according to claim 1, characterized in that: In the step 1, the learnable vector features output by the Transformer decoder in the pre-trained DETR based on the Resnet-101 backbone network are used for comparative learning.
4. The single-stage weakly supervised target localization method for contrastive learning of learnable vector features according to claim 1, characterized in that: In the step 2, 20 learnable vector features with the highest confidence scores are selected as candidate learnable vector features for comparative learning; in the step 3, the text encoder adopts a bidirectional GRU, and the dimensions of the candidate learnable vector features and the text mapping features are set to 512 or 1024; in the step 6, AdamW is used as the optimizer, the batch size is set to 32, the learning rate is set to 1e-4, and the training rounds are set to 25.
5. An electronic device, characterized in that It includes a processor, a memory and an application; the application is stored in the memory and is configured so that the processor executes a single-stage weakly supervised target localization method for comparative learning of learnable vector features as described in any one of claims 1 to 4.
6. A computer-readable storage medium, characterized in that The storage medium stores a computer program; when the computer program is executed in the computer, the computer executes the single-stage weakly supervised target localization method for comparative learning of learnable vector features as described in any one of claims 1 to 4.