Semantic disambiguation-based image-text retrieval method, storage medium and device

By using Faster-RCNN, Bi-GRU, Bert and GAT networks to process ambiguity words in the graphic search model, identifying the context information and semantic annotations of ambiguity words, the comprehension error caused by the context differences between ambiguity words in the prior art is solved, and the accuracy and performance of graphic search is improved.

CN120277230APending Publication Date: 2025-07-08CHONGQING UNIV OF POSTS & TELECOMM
0 Cites 0 Cited by

Patent Information

Application Number
CN202510282112.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-03-11
Publication Date
2025-07-08

AI Technical Summary

Technical Problem

Existing graphic search techniques are difficult to accurately deal with the semantic differences of ambiguity in different contexts, resulting in the model being unable to accurately understand the meaning of text and pictures, especially when the text data is short and the semantic information is limited, the meaning of words and even the entire sentence is misunderstood.

Method used

The pre-trained Faster-RCNN model is used to extract image area features, the Bi-GRU and Bert models encode text data, use the GAT network to update the ambiguity representation, and calculate the meaning similarity between ambiguity words through the relevant layers, generate weight distribution, embed new ambiguity word representations, align with image area features, and adjust model parameters through loss function to achieve accurate recognition and matching of ambiguity words.

Benefits of technology

By identifying the context information of ambiguity and semantic annotations of knowledge sources, the accuracy and performance of graphic and text retrieval is improved, the lack of training data and data distribution dependence is compensated, better ambiguity perception fusion is achieved, and retrieval performance is improved.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The invention belongs to the field of computer vision, and relates to a semantic disambiguity-based image-text retrieval method, a storage medium and a semantic disambiguity-based image-text retrieval device, which comprises the following steps of: obtaining a to-be-retrieved text word, and encoding text data by adopting a Bi-GRU model and a Bert model; carrying out ambiguous word updating on the encoded text by adopting a GAT network; calculating the significance similarity between the updated ambiguous words by adopting a related layer, and generating weight distribution of perceptual significance according to the significance similarity; embedding each ambiguous word into the original text according to the weight distribution to obtain a new ambiguous word expression; aligning the new ambiguous word expression with the image region features; matching the aligned words and images; according to the method, an image-text alignment perception mechanism is designed, and joint optimization of semantic disambiguation and image-text retrieval is achieved. Experiments show that the method significantly improves the accuracy, robustness and practicability of cross-modal image-text retrieval.
Need to check novelty before this filing date? Find Prior Art