Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Image menu retrieval method based on intra-modal and inter-modal hybrid fusion

A modal and recipe technology, applied in the field of cross-modal retrieval, can solve problems such as poor cross-modal retrieval effect

Active Publication Date: 2021-06-08
UNIV OF ELECTRONICS SCI & TECH OF CHINA
View PDF8 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0007] Based on the above problems, the present invention provides an image recipe retrieval method based on intra-modal and inter-modal hybrid fusion, which solves the problem of poor cross-modal retrieval effect

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Image menu retrieval method based on intra-modal and inter-modal hybrid fusion
  • Image menu retrieval method based on intra-modal and inter-modal hybrid fusion
  • Image menu retrieval method based on intra-modal and inter-modal hybrid fusion

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0049] Such as figure 1 The shown image recipe retrieval method based on hybrid fusion within and between modalities includes the following steps:

[0050] Step 1. Prepare image data and recipe data;

[0051] Step 2. Build the overall network;

[0052] Step 3. Train the overall network of step 2, and set the loss function;

[0053] Step 4. Utilize the trained overall network for cross-modal retrieval of food and recipes.

[0054] In the above method, the image data prepared in step 1 includes food images, and the recipe data includes ingredients and cooking steps;

[0055] In this embodiment, step 2 specifically includes the following steps:

[0056] Step 21. extract image feature sequence, food material feature sequence and cooking step feature sequence;

[0057] Step 22. Use the cross-attention mechanism in the recipe mode to perform feature fusion on the ingredient feature sequence and the cooking step feature sequence to obtain the ingredient fusion feature and the cook...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to the field of cross-modal retrieval, in particular to an image menu retrieval method based on intra-modal and inter-modal hybrid fusion, and the method comprises the following steps: 1, preparing image data and menu data; 2, constructing an overall network; 3, training the whole network in the step 2, and setting a loss function; 4, performing cross-modal retrieval on the food and the menu by using the trained overall network; therefore, the problem that the cross-modal retrieval effect is poor is solved.

Description

technical field [0001] The invention relates to the field of cross-modal retrieval, in particular to an image recipe retrieval method based on hybrid fusion within a modality and between modalities. Background technique [0002] With the increasing amount of multimedia data such as text, images, and videos on the Internet, retrieval across different modalities has become a new trend in information retrieval; Semantic-related data; cross-media image-text retrieval is widely aimed at retrieval tasks between images and texts; in the present invention, the application scenario of cross-media image-text retrieval is embodied as the mutual retrieval of food images and recipe texts: pair For any food image, retrieve the recipe description text that is most related to its content description, or for any recipe description text, retrieve the food image that is most related to its description; in general, images and corresponding recipes will be provided in the dataset, where Each re...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/532G06F16/583G06K9/46G06K9/62G06N3/04G06N3/08
CPCG06F16/532G06F16/583G06N3/08G06V10/44G06N3/048G06N3/045G06F18/2415
Inventor 徐行李娇沈复民邵杰申恒涛
Owner UNIV OF ELECTRONICS SCI & TECH OF CHINA
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com