Cross-modal retrieval method and device and storage medium

A cross-modal and modal technology, applied in the field of devices, storage media, and cross-modal retrieval methods, can solve the problem that the cross-modal retrieval model cannot take into account the retrieval accuracy, retrieval speed and model scalability at the same time, achieving The effect of alleviating the semantic gap and improving the accuracy rate

Inactive Publication Date: 2022-08-05
人民中科(北京)智能技术有限公司
View PDF9 Cites 4 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0008] Embodiments of the present disclosure provide a cross-modal retrieval method, device, and storage medium to at least solve the problem that the cross-modal retrieval model in the prior art cannot simultaneously take into account retrieval accuracy, retrieval speed, and model scalability technical issues

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Cross-modal retrieval method and device and storage medium
  • Cross-modal retrieval method and device and storage medium
  • Cross-modal retrieval method and device and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0029] According to this embodiment, a method embodiment of a semantic-based data processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings may be executed in a computer system such as a set of computer-executable instructions, Also, although a logical order is shown in the flowcharts, in some cases, steps shown or described may be performed in an order different from that herein.

[0030] The method embodiments provided in this embodiment may be executed in a mobile terminal, a computer terminal, a server, or a similar computing device. figure 1 A hardware structural block diagram of a computing device for implementing a semantic-based data processing method is shown. like figure 1 As shown, a computing device may include one or more processors (processors may include, but are not limited to, processing means such as a microprocessor MCU or a programmable logic device FPGA, etc.), memory for storing data, and memory ...

Embodiment 2

[0107] Figure 10 The cross-modal retrieval apparatus 1000 according to this embodiment is shown, and the apparatus 1000 corresponds to the method according to the first aspect of Embodiment 1. refer to Figure 10 As shown, the apparatus 1000 includes: a retrieval data receiving module 1010 for receiving the retrieval data and determining the modality of the retrieval data; a feature extraction module 1020 for inputting the retrieval data into a feature extraction unit having at least two feature extraction units model, and extract the feature representation vector of the retrieved data through the feature extraction unit corresponding to the modality of the retrieved data; the query module 1030 is used to traverse the index database according to the feature representation vector, and query a plurality of candidate retrievals related to the retrieved data and a sorting module 1040 for inputting the retrieval data and candidate retrieval results into a similarity calculation m...

Embodiment 3

[0116] Figure 11 The cross-modal retrieval apparatus 1100 according to this embodiment is shown, and the apparatus 1100 corresponds to the method according to the first aspect of Embodiment 1. refer to Figure 11 As shown, the apparatus 1100 includes: a processor 1110; and a memory 1120, connected to the processor 1110, for providing the processor 1110 with instructions for processing the following processing steps: receiving the retrieval data, and determining the modality of the retrieval data; The data is input into a feature extraction model with at least two feature extraction units, and the feature representation vector of the retrieved data is extracted through the feature extraction unit corresponding to the modality of the retrieved data; the index database is traversed according to the feature representation vector, and the query and retrieval are performed. multiple candidate retrieval results related to the data; and input the retrieval data and the candidate ret...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a cross-modal retrieval method and device and a storage medium. The cross-modal retrieval method comprises the following steps: receiving retrieval data, and determining a modal of the retrieval data; inputting the retrieval data into a feature extraction model with at least two feature extraction units, and extracting a feature representation vector of the retrieval data through the feature extraction unit corresponding to the mode of the retrieval data; traversing an index database according to the feature representation vector, and querying a plurality of candidate retrieval results related to the retrieval data; and inputting the retrieval data and the candidate retrieval results into a similarity calculation model with a multi-modal fusion feature extraction unit, performing similarity calculation, and sorting the candidate retrieval results according to the similarity.

Description

technical field [0001] The present application relates to the technical field of cross-modality retrieval, and in particular, to a method, device and storage medium for cross-modality retrieval. Background technique [0002] Currently, cross-modal retrieval is being used more and more. Through the cross-modal retrieval model set in the computing device, the user can perform retrieval by inputting retrieval data of different modalities. For example, when a user wants to retrieve information related to "airplane", he can either input text retrieval data "airplane" to the computing device for retrieval, or input pictures or videos containing airplanes to the computing device for retrieval. Therefore, the computing device performs retrieval according to the text retrieval data "airplane" input by the user, a picture including an airplane, or a video including an airplane, so as to obtain retrieval results related to the topic of "airplane". [0003] The published invention pat...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/903G06V10/74G06V10/80G06K9/62
CPCG06F16/90335G06F18/22G06F18/253
Inventor 阮晓峰王坚李兵余昊楠胡卫明
Owner 人民中科(北京)智能技术有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap