Image-text matching method and device, storage medium and equipment

A matching method and graphic-text technology, applied in the computer field, can solve problems such as difficulty in training object detectors, unsuitable instance features for matching text, and affecting the accuracy of graphic-text matching, so as to achieve the effect of improving accuracy and simplifying the difficulty of implementation

Active Publication Date: 2019-08-20
TENCENT TECH (SHENZHEN) CO LTD
View PDF4 Cites 26 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] When training an object detector, it is necessary to mark the category and location information of all instances in the image on each image, which makes it difficult to train the object detector; in addition, the object detector and the matching model are trained separately, so the object The instance features identified by the detector may not be suitable for the matching model to match the text, thus affecting the accuracy of image-text matching

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Image-text matching method and device, storage medium and equipment
  • Image-text matching method and device, storage medium and equipment
  • Image-text matching method and device, storage medium and equipment

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0030] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the following will further describe the embodiments of the present application in detail in conjunction with the accompanying drawings.

[0031] Visual content recognition and natural language understanding are two major challenges in the field of artificial intelligence. The current popular research direction is to determine the intersection between images and text, and then implement some applications based on this intersection. For example, generating descriptive text based on images, visual question answering, generating images based on text, mutual retrieval between images and text, and so on.

[0032] This application involves the mutual retrieval of images and texts, the main purpose of which is to search for matching images through a given text or query for matching texts through a given image. The following describes several possible applicati...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention discloses an image-text matching method and device, a storage medium and equipment, and belongs to the technical field of computers. The image-text matching method comprises the steps of obtaining an image and a text to be matched; generating a candidate instance feature set according to the image; aggregating candidate instance features in the candidate instance feature set by using a self-attention mechanism to obtain an instance feature set, each instance feature in the instance feature set corresponding to one object in the image; encoding the text to obtaina text vector; and calculating a matching degree between the image and the text according to the instance feature set and the text vector. According to the embodiment of the invention, the realization difficulty of image-text matching can be simplified, and the accuracy of image-text matching can be improved.

Description

technical field [0001] The embodiments of the present application relate to the field of computer technology, and in particular to an image-text matching method, device, storage medium, and equipment. Background technique [0002] Cross-modal retrieval is a new type of retrieval method, which can realize data retrieval between different modalities. Taking image-text mutual retrieval as an example, the user can input an image to retrieve the description text of the image, or the user can input a text to retrieve the image described by the sentence. [0003] Taking text retrieval based on images as an example, the server can generate retrieval results based on the matching degree between the retrieved text and images. When calculating the matching degree between the text and the image, the server uses the trained object detector to extract the instance feature set of the image; uses the cyclic neural network to generate the text vector of the text; uses the matching model to ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/58G06F16/53G06F16/55G06F16/78G06F16/73G06F16/738
CPCY02D10/00
Inventor 贲有成吴航昊袁春周杰
Owner TENCENT TECH (SHENZHEN) CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap