Image reconstruction method and device for cross-modal communication system

A communication system and image reconstruction technology, applied in image data processing, neural learning methods, 2D image generation, etc., can solve problems such as inapplicability, and achieve the goal of bridging gaps, improving quality and category accuracy, and enhancing immersive experience Effect

Active Publication Date: 2021-11-09
NANJING UNIV OF POSTS & TELECOMM
View PDF4 Cites 7 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, these methods require paired data as a training set, and are not suitable for practical application scenarios where it is difficult to collect paired data. Therefore, designing a better method is an urgent problem for relevant workers.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Image reconstruction method and device for cross-modal communication system
  • Image reconstruction method and device for cross-modal communication system
  • Image reconstruction method and device for cross-modal communication system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0090] In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0091] refer to figure 2 , an image reconstruction method and device for a cross-modal communication system, comprising the following steps:

[0092] Step 1. Select the tactile signal and image data pair received by the receiving end of the cross-modal communication system, and use the tactile signal and image data pair as a training set; each tactile signal and image data pair in the training set contains image data Label information of the category to which it belongs;

[0093] Step 2. Establish a cross-modal image generation model based on tactile signals, such as image 3 As shown, the model includes image feature extraction module, cross-modal semantic learning module based on attention mechanism and adversarial image generation module; amon...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses an image reconstruction method for a cross-modal communication system. The method comprises the following steps: reconstructing a missing image in a transmission process through employing a complete tactile signal received by a receiving end in the cross-modal communication system, and constructing a cross-modal interaction network in combination with an attention mechanism. The limitation that a traditional generation model can only train paired samples is solved. The invention further discloses an image reconstruction device for the cross-modal communication system. According to the method, the semantic correlation among different modal data is fully utilized. The cross-modal generation from the tactile signal to the image is realized for unpaired data. The problem that practical training tactile signal-image data is difficult to acquire in pairs in a cross-modal communication system is solved. The quality and the category precision of the generated image are remarkably improved.

Description

technical field [0001] The invention relates to the field of cross-modal image generation, in particular to an image reconstruction method and device for a cross-modal communication system. Background technique [0002] Cross-modal communication is a new type of communication technology that aims at the coordinated transmission and joint processing of visual, audio, and tactile signals. It supports cross-modal multimedia services and has the characteristics of multi-dimensional integration and richer immersive sensory experience. However, in the process of cross-modal communication, there are often problems such as limited transmission capacity and transmission interference, which cause the image data received by the terminal to be blurred or even completely lost. [0003] Aiming at various interference factors in the transmission process, relevant scholars have proposed a series of image restoration methods. For example, in order to remove the impulsive noise in image rest...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06T11/00G06K9/62G06N3/04G06N3/08G06F30/27
CPCG06T11/001G06N3/08G06F30/27G06N3/045G06F18/214G06F18/241G06F18/253G06N3/0464G06N3/0475G06N3/09G06N3/094G06N3/0442G06N3/048G06T11/00G08B6/00
Inventor 魏昕章萌高子涵周亮
Owner NANJING UNIV OF POSTS & TELECOMM
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap