Image-tactile signal mutual reconstruction method and device

Abstract

The invention relates to an image-haptic signal mutual reconstruction method and device. The method comprises the following steps: receiving a signal to be reconstructed, wherein the signal to be reconstructed is a visual signal or an image signal; inputting a signal to be reconstructed into the trained deep reconstruction network model for data reconstruction to obtain a reconstructed signal; the step of training the deep reconstruction network model comprises the following steps: building the deep reconstruction network model formed based on an auto-encoder, dividing a data set into a training set and a test set, and carrying out first-stage and second-stage training on the deep reconstruction network model; the training in the first stage and the training in the second stage are alternately and repeatedly carried out until the training result of the deep reconstruction network model converges, and a preliminary deep reconstruction network model is obtained; and inputting the image-touch signals in the test set into the preliminary deep reconstruction network model for testing to obtain a trained deep reconstruction network model. And the generalization capability and the recovery quality of receiving end signal recovery of the cross-modal reconstruction method are improved.

Description

technical field [0001] The present application relates to the technical field of cross-modality reconstruction, and in particular, to a method and apparatus for mutual reconstruction of image-tactile signals. Background technique [0002] In the modern communication process, audio and visual collaboration provides a better sense of user experience. In order to further realize the immersive communication experience, it is considered to integrate the important perception source of touch and traditional image signals into a new cross-modal communication service. However, it is worth noting that the processing and transmission of haptic signals and traditional image signals are significantly different, and it is challenging to deal with packet loss or corruption. If we can utilize cross-modal prior knowledge to recover lost modalities from non-lost modalities, it can bring great benefits to cross-modal communication, laying the foundation for accurate and reliable immersive comm...

AI Technical Summary

Problems solved by technology

First of all, in the conversion process, most of them directly use the one-to-one pairing of the original modality and the target modality as the model input and output, ignoring the high-level semantic correlation between the modalities; secondly, these generative models all rely on large-scale datasets. Support, as a new data mode, tactile data lacks a large database available, and the form of visual / tactile data pairs that cross-modal reconstruction tasks rely on is even scarcer. Therefore, the current cross-modal reconstruction methods receive Signal recovery has poor generalization and recovery quality

Images