Image description prediction methods for weakly supervised directional visual understanding
By employing a single-stage object detector and anchor text contrast learning method, the problems of slow inference speed and high noise in weakly supervised directional visual understanding are solved, achieving efficient image description prediction and improving detection accuracy and speed.
Patent Information
- Application Number
- CN202310040718.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-01-13
- Publication Date
- 2025-10-31
- Estimated Expiration
- 2043-01-13
AI Technical Summary
Existing technologies suffer from slow inference speed, high noise, and limitations in batch size for negative sample selection in weakly supervised directional visual understanding, which affect detection accuracy.
A single-stage object detector is adopted, and candidate anchor features are selected through multi-scale fusion and confidence filtering. Semantic alignment is performed by using contrast learning between anchors and text, and the loss function is optimized to achieve alignment between image and semantics.
It improves inference speed by 5 times, reduces candidate anchor noise, and enhances detection performance to the state-of-the-art level, solving the detection accuracy problem existing in the prior art.
Smart Images
Figure CN116071544B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing technology, and relates to weakly supervised directional visual understanding, and in particular to an image description prediction method for weakly supervised directional visual understanding. Background Technology
[0002] Reference representation understanding (REC) aims to locate target instances in images based on reference representations. As a cross-modal recognition task, REC is not limited to a fixed set of objects and can theoretically perform detection of any open set. These attractive properties have made it increasingly popular in industry and academia. However, the expensive instance-level annotation has long limited its development.
[0003] Most current methods and literature extend two-stage object detectors, such as Faster R-CNN, to the weakly supervised REC task. They treat the REC task as a region-based text ranking problem, where salient regions of an image are first extracted by Faster R-CNN, and then ranked by cross-modal matching. To achieve weakly supervised training, they use only expressions as supervision information and optimize the ranking matching module through semantic reconstruction or cross-modal contrastive learning. However, due to the inherent characteristics of Faster R-CNN, these methods tend to have poor inference speed, similar to existing two-stage REC models. Summary of the Invention
[0004] The purpose of this invention is to address the aforementioned problems in the existing technology by providing an image description prediction method for weakly supervised directional visual understanding.
[0005] To achieve the above objectives, the technical solution of the present invention is as follows:
[0006] An image description prediction method for weakly supervised directional visual understanding is proposed. This method trains a model based on an RGB image and a descriptive language, without ground truth bounding boxes, to locate the object referred to by the descriptive language within the RGB image. The method includes the following steps:
[0007] Step 1: Set the size of the input RGB image to 416×416×3, and set the maximum text input for the description language to 15.
[0008] Step 2: Obtain the visual features of the RGB image at three scales respectively;
[0009] Step 3: The visual features at the three scales are fused through multi-scale fusion. The visual features are then selected by scale selection and confidence selection, and the 13×13 scale features are selected. Based on the box confidence, 90% of the low-confidence features are filtered out to obtain candidate anchor point features.
[0010] Step 4: Project the candidate anchor features and text features onto the same dimension using a fully connected layer to obtain... and
[0011] Step 5: During the training process, for a batch of image-text pairs, given text, calculate the similarity between candidate anchors and corresponding texts in different images. For matching image-text pairs, select the anchor text pair with the highest similarity as a positive sample. For mismatched image-text pairs, select the top two anchor text pairs with the highest similarity as negative samples. The alignment between semantics and text is achieved through comparative learning between anchor texts.
[0012] Step 6: In the prediction process, given an image-text pair, calculate the similarity between the candidate anchors in the text and the image, select the anchor with the highest similarity, obtain the corresponding predicted box based on its index, and output the box with the highest confidence as the target box.
[0013] In step 2, the acquisition of visual features of the RGB image at three scales is achieved using a YOLOv3 encoder trained on the COCO object detection dataset as the visual encoder to obtain the visual features of the RGB image at three scales: And the corresponding box, respectively denoted as Where, N a This represents the number of predefined boxes for each anchor point. The C code records the predicted box results, including confidence information, box offset, and linguistic features describing the language.
[0014] In step 3, the visual features are successively processed through scale selection and confidence selection. The formula for calculating the anchor feature confidence is as follows:
[0015]
[0016] Where i∈[0,13], j∈[0,13], N a This represents the number of predefined boxes for each location.
[0017] In step 5, semantic and text alignment is achieved through contrastive learning between anchor texts. Contrast learning is performed between anchors and text, and negative samples are augmented through similarity-based sampling. The loss function is calculated as follows:
[0018]
[0019] in, These are the anchor points sampled from the batch. τ represents the positive samples for image-text pair i; N and M represent the number of negative anchor points and batch size for each image, respectively; τ is the temperature coefficient.
[0020] During the training of the neural network, the Adam optimizer was used, with the initial learning rate and batch size set to 0.0001 and 64, respectively.
[0021] This invention utilizes contrastive learning between anchor points and text for weakly supervised directional visual understanding tasks. Using single-stage object detection for task modeling can lead to problems such as overly dense predicted bounding boxes and excessive noise; furthermore, the selection of negative samples in traditional contrastive learning is often limited by the batch size, all of which affect detection accuracy.
[0022] This invention uses a pre-trained YOLOv3 backbone network to obtain visual features at three scales, namely anchor features and their corresponding predicted bounding boxes, from RGB images. These three scale visual features are first fused at multiple scales, then filtered by scale and confidence to obtain candidate anchor features. Finally, the similarity between the candidate anchor features and their corresponding text features is calculated. During training, the optimization objective is to maximize the highest similarity score between anchor features and corresponding text in matched image-text pairs, and minimize the similarity score between anchor features and text in mismatched image-text pairs, thereby achieving image-semantic alignment in the absence of true bounding box annotations. During prediction, the anchor feature with the highest similarity to the text is selected, and its corresponding predicted bounding box is found based on its index. The predicted bounding box with the highest confidence is selected as the target bounding box for output.
[0023] Compared with the prior art, the innovation of this invention is reflected in:
[0024] (1) This invention proposes a novel single-stage contrastive model to achieve weakly supervised REC. By using single-stage modeling, we do not need the time-consuming region of interest pooling and non-maximum suppression operations in the background technology, thereby significantly improving the inference speed by 5 times.
[0025] (2) This invention uses a single-stage target detector for modeling, redefining the weakly supervised REC task as an anchor selection problem. This invention employs scale selection and confidence filtering methods to effectively reduce the number of candidate anchors and reduce noise; and expands the negative samples by sampling multiple anchors based on similarity, so that it is not limited by the batch size, thereby effectively improving the detection performance of the model and reaching the state-of-the-art level. Attached Figure Description
[0026] Figure 1 This is a schematic diagram of the network structure of the present invention.
[0027] Figure 2 This is a diagram illustrating anchor point selection. Detailed Implementation
[0028] The technical solution and beneficial effects of the present invention will be described in detail below with reference to the accompanying drawings.
[0029] This invention provides an image description prediction method for weakly supervised directional visual understanding, comprising the following steps:
[0030] I. Model Implementation Process:
[0031] 1.1 Model Input:
[0032] The model takes an RGB image of size 416×416×3 as input and a descriptive text for an object in the image. The maximum text input value is set to 15. Figure 1 As shown.
[0033] 1.2 Visual Feature Encoder:
[0034] For the input RGB image, using COCO [1] Pre-trained YOLOv3 on object detection datasets [2] This serves as a visual encoder, thereby obtaining visual features at three scales: And the corresponding box, respectively denoted as Where, N a This represents the number of predefined boxes for each anchor point. C records the prediction results of the boxes, including confidence information and box offset.
[0035] 1.3 Language Feature Encoder:
[0036] For the input description language, we first utilize the pre-trained word vectors GloVe. [3] To convert the language into feature vectors, and then use LSTM [4] As a language encoder, we obtain language features:
[0037] 1.4 Target Anchor Point Feature Selection:
[0038] like Figure 2 As shown, in order to obtain sparser candidate anchor point features, the visual features are scaled to 13×13. Then, for each anchor point, the average confidence of its corresponding predicted box is calculated as the confidence of that point. In form:
[0039]
[0040] Where i∈[0,13], j∈[0,13], N aThis represents the number of boxes at each location. Based on confidence information, the top 10% are selected, and 90% of the anchors located at low-confidence locations are filtered out to obtain candidate anchors. The corresponding visual features are denoted as […]. Where K = 17. The similarity between the candidate anchor point features and the text features is calculated using the following formula, and the anchor point with the highest similarity is selected as the target anchor point:
[0041] sim(f a ,f t )=(f a W a ) T (f t W t ),
[0042] Among them, W a and W t It is the corresponding projection matrix.
[0043] 1.5 Target box output:
[0044] After calculating the similarity between candidate anchors in the text and image, the anchor with the highest similarity is selected, and the corresponding predicted box is obtained according to its index. The box with the highest confidence is then output as the target box.
[0045] II. Model Training Process:
[0046] 2.1 Calculation of the loss function:
[0047] The candidate anchor features and text features obtained in step one are used to model the weakly supervised directional visual understanding task.
[0048] Specifically, cross-modal contrastive learning is used between anchor points and text. For a batch of image-text pairs, given text, the similarity between candidate anchor points and corresponding text in different images is calculated. For matching image-text pairs, the anchor-text pair with the highest similarity is selected as a positive sample. For mismatched image-text pairs, the top two anchor-text pairs with the highest similarity are selected as negative samples. The loss function used is calculated as follows:
[0049]
[0050] in, These are the anchor points sampled from the batch. For image-text pair i, these are the positive samples. N and M represent the number of negative anchors and the batch size for each image, respectively. τ is the temperature coefficient.
[0051] Gradient descent can be used to optimize the parameters of the trained model.
[0052] 2.2 Model training parameter settings:
[0053] During training, the Adam optimizer was used, with an initial learning rate of 0.0001 and a batch size of 64. The model was trained for a total of 25 epochs.
[0054] III. Model Deployment Process:
[0055] After model training in step two, the trained model can be used to obtain the model's output, i.e., the target bounding box, as the model's prediction result by inputting the image and description language in step one.
[0056] References:
[0057] [1]Tsungyi Lin, Michael Maire, Serge J Belongie, James Hays, PietroPerona, Deva Ramanan, Piotr Dollar, and C Lawrence Zitnick. Microsoft coco: Commonobjects in context. In ECCV, 2014.
[0058] [2]Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. InarXiv preprint, 2018.
[0059] [3]Jeffrey Pennington,Richard Socher,and Christopher Manning.Glove:Global vectors for word representation.In EMNLP,2014.
[0060] [4]Sepp Hochreiter and Jurgen Schmidhuber.1997.Long short-termmemory.Neural Computation(1997).
[0061] The above embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solutions based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.
Claims
1. An image description prediction method for weakly supervised directional visual understanding, characterized in that... Includes the following steps: Step 1: Set the size of the input RGB image to 416×416×3, and set the maximum text input for the description language to 15. Step 2: Obtain the visual features of the RGB image at three scales and the linguistic features of the descriptive language, respectively; Step 3: The visual features at the three scales are fused through multi-scale fusion. The visual features are then selected by scale selection and confidence selection, and the 13×13 scale features are selected. Based on the box confidence, 90% of the low-confidence features are filtered out to obtain candidate anchor point features. Step 4: Project the candidate anchor features and text features onto the same dimension using a fully connected layer to obtain... and Step 5: During the training process, for a batch of image-text pairs, given text, calculate the similarity between candidate anchors and corresponding texts in different images. For matching image-text pairs, select the anchor text pair with the highest similarity as a positive sample. For mismatched image-text pairs, select the top two anchor text pairs with the highest similarity as negative samples. The alignment between semantics and text is achieved through comparative learning between anchor texts. The method achieves semantic and text alignment through contrastive learning between anchor texts. It uses contrastive learning between anchors and text, and simultaneously augments negative samples through similarity-based sampling. The loss function used is calculated as follows: in, These are the anchor points sampled from the batch. For image-text pair i, is the positive sample; N and M represent the number of negative anchor points and batch size for each image, respectively; τ is the temperature coefficient. Step 6: In the prediction process, given an image-text pair, calculate the similarity between the candidate anchors in the text and the image, select the anchor with the highest similarity, obtain the corresponding predicted box based on its index, and output the box with the highest confidence as the target box.
2. The image description prediction method for weakly supervised directional visual understanding as described in claim 1, characterized in that... In step 2, the acquisition of visual features of the RGB image at three scales is achieved using a YOLOv3 encoder trained on the COCO object detection dataset as the visual encoder to obtain the visual features of the RGB image at three scales: And the corresponding box, respectively denoted as Where, N a This represents the number of predefined boxes for each anchor point. The C code records the predicted box results, including confidence information, box offset, and linguistic features describing the language.
3. The image description prediction method for weakly supervised directional visual understanding as described in claim 1, characterized in that... In step 3, the visual features are successively processed through scale selection and confidence selection. The formula for calculating the anchor feature confidence is as follows: Where i∈[0,13], j∈[0,13], N a This represents the number of predefined boxes for each location.
4. The image description prediction method for weakly supervised directional visual understanding as described in claim 1, characterized in that... In step 5, during the training process, the Adam optimizer is used, and the initial learning rate and batch size are set to 0.0001 and 64, respectively.