Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Fine-grained cross-media retrieval method based on unified double-branch network

A branch network and cross-media technology, applied in the field of fine-grained cross-media retrieval based on a unified dual branch network, to achieve the effects of accurate semantic feature representation, low computational cost, and reduced heterogeneity differences

Inactive Publication Date: 2021-12-10
南京码极客科技有限公司
View PDF2 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

The performance indicators shown in the experiment prove the effectiveness of the method, but because the method ignores the unique information of the media to a certain extent, there is still a lot of room for improvement in the retrieval performance

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Fine-grained cross-media retrieval method based on unified double-branch network
  • Fine-grained cross-media retrieval method based on unified double-branch network
  • Fine-grained cross-media retrieval method based on unified double-branch network

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0043] This embodiment proposes a fine-grained cross-media retrieval method based on a unified dual-branch network, such as figure 1 shown, including the following steps:

[0044] Step 1: use the sample training set to train a unified double-branch network model; the unified double-branch network model includes a unified convolutional neural network feature extraction module, a unified probability feature branch, a media independent feature branch and a cross-media public feature combination module; The unified convolutional neural network features are respectively connected with the unified probability feature branch and the media independent feature branch to extract a unified common convolution feature ; The output ends of the unified probability feature branch and the media independent feature branch are respectively connected with the media independent feature branch; learning; the media-independent feature branch is used to learn the corresponding media types respecti...

Embodiment 2

[0049] In this embodiment, on the basis of the above-mentioned embodiment 1, in order to better realize the present invention, further, the specific operation of the step 3 is as follows:

[0050] Step 3.1: Extract the common convolutional features of the input samples through the unified convolutional neural network feature extraction module ;

[0051] Step 3.2: Combine the obtained common convolutional features Input to the unified probability feature branch and the media independent feature branch respectively;

[0052] Step 3.3: Co-convolve features via the unified probabilistic feature branch process to obtain the uniform probability feature of the input sample ; common convolutional features via media-independent feature branches process to obtain the media-independent features of the input samples ;

[0053] Step 3.4: Set probability correction features ; The probability correction feature Unified probabilistic features for and across media A vector of...

Embodiment 3

[0059] In this embodiment, on the basis of any one of the above-mentioned embodiments 1-2, in order to better realize the present invention, further, the specific operation of the step 3.6 is as follows:

[0060] Step 3.6.1: Use each of the K input samples to correct the features for the probability Update, the specific update operation is: through the same media similarity measurement, obtain the category label of the training sample that is most similar to the input in the database, and modify the probability correction feature The probability value corresponding to the category of a certain real label pair, and the category probability value of a certain real label pair The specific update formula is as follows:

[0061] ;

[0062] Step 3.6.2: Correct the feature with the updated correction probability with uniform probability features weighted combination to get the final cross-media public features , the specific weighted combination formula is as follows:

...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a fine-grained cross-media retrieval method based on a unified double-branch network. The method adopts a unified deep convolutional neural network structure to extract preliminary public features, and then corrects media independent features lacked by the public features through media exclusive branches with a relatively low calculation amount. For a unified probability feature branch, a cross entropy loss is uniformly used to learn probability features without distinguishing media types. For media feature branches, independent features of various media types are obtained by using a dedicated module for each media type. And then, the probability feature is combined with the media feature to obtain a final public feature, and the final public feature is used in a retrieval process. According to the network structure, a unified convolutional network is adopted as a trunk, the calculation cost is relatively low, meanwhile, the independent feature of each media type is considered, and the features of various media types can be effectively extracted.

Description

technical field [0001] The invention belongs to the technical field of computer deep neural network learning, and in particular relates to a fine-grained cross-media retrieval method based on a unified dual-branch network. Background technique [0002] In recent years, the common space learning method based on deep neural network is the most commonly used method in the field of cross-media retrieval. The input data of different media types is mapped to the common feature space through the deep neural network, and the retrieval results are generated according to the similarity ranking between the common features corresponding to the input samples and the candidate objects in the data set. To obtain a common feature space, the methods are generally divided into two categories, media-specific network-based methods and media-unified network-based methods. Media-specific networks build their dedicated networks for each media type, and these dedicated networks have different netw...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/43G06F16/483G06K9/62G06N3/04G06N3/08
CPCG06F16/43G06F16/483G06N3/08G06N3/045G06F18/22G06F18/214
Inventor 沈复民姚亚洲孙泽人陈涛张传一
Owner 南京码极客科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com