An image super-resolution reconstruction method for cross-modal communication

Abstract

The application belongs to the technical field of super-resolution reconstruction of visual signals, and discloses an image super-resolution reconstruction method for cross-modal communication, which first transmits only low-resolution visual signals and corresponding tactile signals at a sending end, then compensates for semantic gaps existing between different modes through learning of intra-modal discriminativeness and inter-modal correlation, realizes complementary learning at a receiving end through an effective feature fusion mode after channel transmission, and finally generates a required high-resolution visual signal by using obtained fusion features. The application solves the problem of visual quality decline caused by limited bandwidth and competition between multi-modal signals in multi-modal services, and finally affects user experience, realizes consistent and complementary learning of cross-modal signals, guarantees acquisition of high-resolution visual signals at the receiving end under limited bandwidth, and improves immersive experience of users.

Description

Technical Field

[0001] This invention belongs to the field of super-resolution reconstruction technology of visual signals, specifically relating to an image super-resolution reconstruction method for cross-modal communication. Background Technology

[0002] With the rapid development of wireless and multimedia communication technologies, human audiovisual needs have been greatly satisfied, leading to a pursuit of more diverse and richer experiences. When tactile signals are combined with traditional audiovisual signals, multimodal services emerge, offering more granular interaction and immersive experiences. Multiple studies have found that in online multimodal service scenarios, people can enhance their perception and interactive experience of products through high-resolution visual signals and high-fidelity tactile signals. For example, in online shopping, consumers can obtain detailed information about product details, internal texture, hardness, and other characteristics through touch and observation. To support multimodal services, cross-modal communication has emerged, ensuring coordinated transmission and processing between multiple modalities by leveraging the correlations between different modalities. However, limited bandwidth and intermodal competition make existing cross-modal communication schemes difficult to implement, leading to a decline in user immersion, particularly unsatisfactory visual experiences.

[0003] Specifically, on the one hand, high-fidelity visual signals are an important guarantee for users' immersive experience. However, due to limited bandwidth, it is difficult to transmit such high-resolution images / videos in online multimedia communication services. On the other hand, visual and tactile modalities compete during transmission. In order to meet the requirements of low latency and high reliability of tactile signals, existing solutions usually give tactile signals a higher priority. However, the frequent and irregular occurrence of tactile signals will seriously affect the transmission quality of visual signals, especially when users have a need for frequent touching, such as online shopping.

[0004] Currently, the problem of visual signal quality degradation caused by insufficient bandwidth and intermodal competition can be addressed through two approaches: multimodal communication and super-resolution reconstruction. Multimodal communication solutions mainly include traditional audio-visual communication solutions and haptic communication solutions. These solutions can achieve high-fidelity transmission of audio-visual or haptic signals independently, but they cannot guarantee the quality of the receiving end when scenarios involving simultaneous transmission of audio-visual and haptic signals are involved. Super-resolution reconstruction solutions mainly utilize low-resolution vision to reconstruct high-resolution visual signals based on a single visual signal or based on reference information (such as visual signals from different angles, adjacent frames, boundary maps, etc.). However, they mostly complete the reconstruction task at the local terminal and do not involve communication tasks.

[0005] The existing multimodal transmission schemes for limited bandwidth have the following drawbacks: they consider each mode separately and fail to make reasonable use of the consistency and complementarity between multimodal data; they do not consider the communication process and only process data at the terminal, without considering the competition between modes that exists during multimodal data transmission. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, this invention provides an image super-resolution reconstruction method for cross-modal communication. It relies on the semantic consistency between tactile and visual signals, fully considers the intra-modal and inter-modal relationships to extract and map the features of each modality, and then uses a powerful feature fusion network to effectively generate fused features that are as similar as possible to the features of high-resolution visual signals by using low-resolution visual and tactile signals, and finally obtains high-resolution visual signals, thus ensuring the immersive user experience in bandwidth-constrained multimodal application scenarios.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0008] This invention is an image super-resolution reconstruction method for cross-modal communication, comprising the following steps:

[0009] Step (1): Using the complete high-resolution visual signal, encode and decode the high-resolution visual signal. Train the encoding network of the high-resolution visual signal through the encoding step and obtain the encoding features of the high-resolution visual signal. Train the decoding (generation) network of the high-resolution visual signal through the decoding step to provide support for the subsequent visual signal super-resolution reconstruction model.

[0010] Step (2): Design a haptic-assisted visual signal super-resolution reconstruction (HaSR) model; the HaSR super-resolution reconstruction model is as follows:

[0011] After acquiring visual and tactile signals from the terminal, the visual signals are first downsampled at the edge nodes to obtain low-resolution visual signals; then, a pre-trained, widely used coding network is used to perform preliminary feature extraction on the low-resolution visual signals and the corresponding tactile signals.

[0012] Then, by leveraging intramodal discriminativeness and intermodal consistency, a mapping network is used to reduce intermodal differences and mine the correlations between different modes to bridge the semantic gap between modes. Thus, based on the preliminary features extracted by encoding, mapping features with semantic discriminativeness and semantic association are obtained. The obtained mapping features are then normalized and passed through a channel model for use in the next step of feature fusion.

[0013] Based on the mapping features of low-resolution visual signals and tactile signals, as well as the encoding features of the obtained high-resolution visual signals, and combined with the powerful data fitting ability of generative adversarial networks, fused features are obtained.

[0014] Finally, the obtained fused features are input into the high-resolution visual signal generation network to achieve the reconstruction of the high-resolution visual signal;

[0015] Step (3): Use optimization methods to train the HaSR model and finally obtain the optimal model parameters for the subsequent testing phase.

[0016] Step (4): Input the pair of low-resolution visual signals and tactile signals to be tested into the optimal HaSR model. The optimal HaSR model is used to extract the features of the low-resolution visual signals and the corresponding tactile signals and fuse them. The fused features are then used to generate the required high-resolution visual signals.

[0017] A further improvement of the present invention is that step (1) includes the following steps:

[0018] (1-1) For the training dataset It contains paired tactile, low-resolution visual, and high-resolution visual signals, where N is the number of paired visual and tactile signals, and d i ={h i ,l i ,t i} represent high-resolution visual signals, low-resolution visual signals, and the corresponding tactile signals, respectively. The i-th high-resolution visual signal h... i Input a high-resolution visual signal into an encoding network, and extract the encoded features z of the visual signal. h ;

[0019] (1-2) Encode the features z of the obtained high-resolution visual signal h The input is fed into a decoder of a high-resolution visual signal composed of a generative adversarial network (GAN). The reconstructed high-resolution visual signal is then input into a discriminator of the high-resolution visual signal. By jointly training the encoder and decoder and optimizing them using reconstruction loss and discriminative loss, the encoded features of the high-resolution visual signal and the corresponding decoding network are finally learned. Specifically, the defined loss function is as follows:

[0020] L pre =L rec +αL pre-adv ,

[0021] Here, α is a coefficient used to adjust the proportion of different losses, and the first loss term is the reconstruction loss:

[0022]

[0023] Where (C, H, W) is the size of the high-resolution visual signal, G h Represents the decoding (generation) network of the corresponding high-resolution visual signal. The first term represents the encoded features of the high-resolution visual signal obtained by the corresponding encoding network, and ||·||1 represents the corresponding L1 norm; the second term is the loss of the generative adversarial network, and the specific loss function is defined as follows:

[0024]

[0025] Where E(*) represents the expected value of the distribution function, p(z) h ) represents the distribution of features in the high-resolution image, and p(h) represents the distribution of features in the true high-resolution image, where D h A discrimination network representing the corresponding high-resolution visual signal is used to judge the reconstructed high-resolution visual signal. h A discrimination network representing the corresponding high-resolution visual signal is used to judge the reconstructed high-resolution visual signal, θ. gh and θ dh Let L represent the parameters of the generator and discriminator for the corresponding high-resolution visual signals, respectively. pre The optimal encoding network for high-resolution visual signals, the corresponding encoding features of high-resolution visual signals, the corresponding decoding network (i.e., the generator network), and the corresponding discrimination network for high-resolution visual signals are obtained.

[0026] A further improvement of the present invention is that step (2) includes the following steps:

[0027] (2-1) Preliminary feature extraction of low-resolution visual signals and corresponding tactile signals, based on The paired low-resolution visual signals present in i and tactile signal h i Using existing mature deep neural networks, preliminary feature extraction of low-resolution visual and tactile signals is performed to obtain the corresponding encoded features f of the low-resolution visual signals. l and the corresponding tactile signal encoding features f t ;

[0028] (2-2) Based on the coding features f of the obtained low-resolution visual signal l and the corresponding tactile signal encoding features f tA feature mapping network is established to effectively reduce intermodal heterogeneity. It learns intramodal discriminative representations and intermodal consistent representations from both intramodal and cross-modal perspectives, ultimately obtaining mapped features, including the mapped features z of low-resolution visual signals. l Mapping characteristics z of tactile signals t Then the obtained mapping features are input into the channel model, and the corresponding feature fusion steps are executed after the main terminal receives them.

[0029] Cross-modal semantic relevance learning: Triple loss is used for cross-modal semantic relevance learning. After learning through the mapping network, the following effect is achieved: for low-resolution visual and tactile signal features from the same category, the distance between them should be close; for low-resolution visual and tactile signal features from different categories, the distance between them should be far. The loss function is defined as follows:

[0030]

[0031]

[0032] Where θ l and θ t Let p and q represent the parameters of the corresponding low-resolution visual signal mapping network and tactile signal mapping network, respectively. p and q represent different categories, N represents the number of instances of the corresponding low-resolution visual signal and tactile signal, σ represents the corresponding threshold, and L2 = ||·||2 represents the corresponding L2 norm. The total loss of semantic relevance can be expressed as the sum of the above two, i.e.:

[0033]

[0034] Discriminative learning within the same modality: While ensuring semantic relevance, it is necessary to effectively solve the semantic discrimination problem within the same modality. That is, for samples within the same modality (visual modality or tactile modality), samples belonging to the same category should be closer together, and samples belonging to different categories should be farther apart. This is mainly achieved by adding a common classifier after the mapping network. The specific loss is expressed as follows:

[0035]

[0036] Where, p i (z) represents the probability distribution predicted by the classifier, y i It's a real label, θ c The parameters represent the corresponding common classifier; after the above processing, the mapping features z of the obtained low-resolution visual signal are... l Mapping characteristics z of tactile signals tThe input channel model is normalized;

[0037] (2-3) After transmission through the channel, the main terminal obtains the mapping features z of the corresponding noisy, low-resolution visual signal. l-n Mapping characteristics z of tactile signals t-n The purpose of this step is to utilize the complementarity of the two to generate fusion features, so that the generated fusion features are as similar as possible to the encoded features z of the high-resolution visual signal in step (1). h Similarly, based on this objective, generative adversarial networks (GANs) are chosen to perform the feature fusion task by leveraging their ability to fit data distributions; where z h z is considered a real sample l-n and z t-n z is considered as input to the generator m The loss of the fusion network, representing the obtained fusion features, is defined as follows:

[0038] L m =L m-adv (G m D m )+L2(z m ,z h ),

[0039] The first term represents the loss of a typical generative adversarial network, G. m Representative feature fusion network, D m This represents the feature discrimination network corresponding to the feature fusion network, specifically expressed as:

[0040]

[0041] Where θ gm θ represents the corresponding parameter of the fusion network generator. dm The first term represents the model parameters of the discriminant network corresponding to the fusion network generator; the second term represents the L2 loss, which helps maintain semantic consistency.

[0042] (2-4) After obtaining the fusion features, the high-resolution visual signal is reconstructed using these features. The high-resolution visual signal decoding (generation) network and the corresponding discrimination network obtained in the first step are used, and their network structure and parameters are used as the initial network parameters for this step. Furthermore, a perceptual loss is added to the previous loss to make the generated visual signal more consistent with human perceptual characteristics. The specific loss in this step is as follows:

[0043] L finet =L per +βL adv-finet +γL rec ,

[0044] The first term is the perceptual loss, which can be specifically expressed as:

[0045]

[0046] Where M i,j F represents the number of parameters in the corresponding feature map. i,j The first term represents the output after the i-th convolutional layer and before the j-th max-pooling layer of the VGG-19 network, and the second term represents the loss of the network generating the high-resolution visual signal, which can be specifically expressed as:

[0047]

[0048] Where θ gh This represents the corresponding utilization of fusion feature z m The parameters of the generative network used for high-resolution visual signal reconstruction, θ dh The first term represents the parameters of the corresponding discrimination network, the third term is the reconstruction loss; in addition, β and γ are hyperparameters.

[0049] A further improvement of the present invention is that step (3) includes the following steps:

[0050] (3-1) Using existing high-resolution visual signals, train the encoding network and corresponding decoding (generation) network for the high-resolution visual signals, as well as the discrimination network for the corresponding high-resolution visual signals. The specific process is as follows:

[0051] Step 311: Initialize parameter θ eh (0), θ gh (0), θ dh (0) represents the value of the corresponding parameter in the 0th iteration;

[0052] Step 312: Set the number of iterations to n1 and the learning rate to μ1;

[0053] Step 313: Optimize network parameters using the RMSProp algorithm.

[0054]

[0055]

[0056]

[0057] Where θ eh (n+1), θ dh (n+1), θ gh (n+1) and θ eh (n), θ dh (n), θ gh(n) are the parameters of the encoding network, discrimination network, and generation network of the high-resolution visual signal corresponding to the (n + 1)-th and n-th times respectively. is to take the partial derivative of each loss function;

[0058] Step 314: If n < n1, then repeat Step 313. After n1 rounds of iteration, a network that converges to the optimal is obtained, including the encoding network of the high-resolution visual signal, the discrimination network of the high-resolution visual signal, and the generation network of the high-resolution visual signal.

[0059] (3-2) Based on the encoding network of the high-resolution visual signal, the decoding (generation) network of the high-resolution visual signal, and the corresponding discrimination network obtained in the first step, use the low-resolution visual signal and the corresponding tactile signal to train the encoding and mapping network of the corresponding low-resolution visual signal, the encoding and mapping network of the tactile signal, and the feature fusion network, and fine-tune the generation network of the corresponding high-resolution visual signal. The low-resolution visual signal encoding network and the encoding network of the tactile signal are obtained by loading the pre-trained network and do not participate in the update in this step. The specific process is as follows:

[0060] Step 321: Initialize the parameter θ l (0), θ t (0), θ c (0), which represents the initial random parameter of the corresponding network, and load the parameters of the generation network and discrimination network of the high-resolution visual signal obtained in Step 3-1 as the initial parameters of θ gh (0), θ dh (0).

[0061] Step 322: Start iteration, set the number of iterations to n2, and the learning rate to μ2.

[0062] Step 323: Use the Adam algorithm to optimize the parameters of the low-resolution visual signal mapping network, the tactile signal mapping network, and the common classifier:

[0063]

[0064]

[0065]

[0066] where θ c (n + 1), θ l (n + 1), θ t (n + 1) and θ c (n), θ l (n), θ t(n) are the parameters of the common classifier corresponding to the (n + 1)-th and n-th times respectively, the low-resolution visual signal mapping network and the corresponding tactile signal mapping network, is to take the partial derivative of each loss function;

[0067] Step 324: Adopt the Adam algorithm to optimize the parameters of the feature fusion network and the discriminative network of the corresponding fused features:

[0068]

[0069]

[0070] where θ gm (n + 1), θ dm (n + 1), θ gm (n) and θ dm (n) are the parameters of the generation network of the fused features and the discriminative network of the fused features corresponding to the (n + 1)-th and n-th times respectively, is to take the partial derivative of each loss function;

[0071] Step 325: Adopt the RMSProp algorithm to finely tune the generation network and the discriminative network of the high-resolution visual signal, and optimize and update through the following function:

[0072]

[0073]

[0074] where θ gh (n + 1), θ dh (n + 1), θ gh (n) and θ dh (n) respectively represent the parameters corresponding to the generation network and the discriminative network of the high-resolution visual signal corresponding to the (n + 1)-th and n-th times, is to take the partial derivative of each loss function;

[0075] Step 326: If n < n2, then jump to Step 323. After n2 rounds of iteration, obtain the converged optimal HaSR network, including the low-resolution visual signal mapping network, the tactile signal mapping network, the feature fusion network and the corresponding discriminative network of the fused features, and the finely tuned and converged optimal discriminative network of the high-resolution visual signal, the generation network of the high-resolution visual signal.

[0076] A further improvement of the present invention is that step (4) includes the following steps:

[0077] (4-1) Adopt the trained HaSR model;

[0078] (4-2) Input a set of paired low-resolution visual signals and corresponding tactile signals into the HaSR model to complete the encoding, mapping and fusion of modal features, and finally obtain the corresponding high-resolution visual signals.

[0079] The beneficial effects of this invention are:

[0080] This invention overcomes the ill-posed problem caused by traditional single-modal visual signal super-resolution reconstruction by matching tactile signals and visual signals, i.e., a low-resolution visual signal may correspond to multiple high-resolution visual signals.

[0081] This invention overcomes the heterogeneity differences between different modes by fully exploring the intramodal and intermodal relationships of visual and tactile signals;

[0082] This invention relies on an effective feature fusion method that can fully utilize the complementarity of different modalities to improve the quality of the generated high-resolution visual signal. Attached Figure Description

[0083] Figure 1 This is a flowchart of the image super-resolution reconstruction method of the present invention.

[0084] Figure 2 This is a schematic diagram of the complete network structure of the present invention.

[0085] Figure 3 This is a super-resolution reconstruction result image comparing the present invention with other comparative methods. Detailed Implementation

[0086] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0087] This invention provides an image super-resolution reconstruction method for cross-modal communication, the flowchart of which is shown below. Figure 1 As shown, the method includes the following steps:

[0088] Step 1: Encode and decode the high-resolution visual signal using the complete high-resolution visual signal, i.e. Figure 2 In Phase 1, the encoding network for high-resolution visual signals is trained through the encoding step to obtain the encoded features of the high-resolution visual signals. Then, the decoding (generation) network for high-resolution visual signals is trained through the decoding step to provide support for the subsequent super-resolution reconstruction model of visual signals.

[0089] Step 1-1: For the training dataset It contains paired tactile, low-resolution visual, and high-resolution visual signals, where N is the number of paired visual and tactile signals, and d i ={hi ,l i , t i} represent the high-resolution visual signal 128×128, the low-resolution visual signal 32×32, and the corresponding tactile signal, respectively. The spectrum obtained through short-time Fourier transform is used to represent the i-th high-resolution visual signal h. i An encoding network is input to the high-resolution visual signal. The encoding network consists of VGG-16 pre-trained on ImageNet, followed by fully connected layers with dimensions of 512 and 128, to extract the encoded features z of the visual signal. h Its dimensions are 128;

[0090] Step 1-2: Encode the features z of the obtained high-resolution visual signal h The input is fed into a decoder of a high-resolution visual signal composed of a generative adversarial network (GAN). This decoder consists of a deconvolutional network with 3×3 kernels and corresponding filter sizes of 512, 256, 128, 64, and 3. The reconstructed high-resolution visual signal is then input into a discriminator, which is a stacked convolutional network with 3×3 kernels and corresponding filter data sizes of 64, 128, 256, and 512. Fully connected layers with dimensions of 512, 128, and 1 are then added. The encoder and decoder are jointly trained, and optimization is achieved using reconstruction loss and discriminative loss. Ultimately, the encoded features of the high-resolution visual signal and the corresponding decoding network are learned. Specifically, the defined loss function is as follows:

[0091] L pre =L rec +αL pre-adv ,

[0092] Here, α is a coefficient used to adjust the proportion of different losses, and the first loss term is the reconstruction loss:

[0093]

[0094] Where (C, H, W) is the size of the high-resolution visual signal, G h Represents the decoding (generation) network of the corresponding high-resolution visual signal. The first term represents the encoded features of the high-resolution visual signal obtained by the corresponding encoding network, and ||·||1 represents the corresponding L1 norm; the second term is the loss of the generative adversarial network, and the specific loss function is defined as follows:

[0095]

[0096] Where E(*) represents the expected value of the distribution function, p(z) h) represents the distribution of features in the high-resolution image, and p(h) represents the distribution of features in the true high-resolution image, where D h A discrimination network representing the corresponding high-resolution visual signal is used to judge the reconstructed high-resolution visual signal. h A discrimination network representing the corresponding high-resolution visual signal is used to judge the reconstructed high-resolution visual signal, θ. gh and θ dh Let L represent the parameters of the generator and discriminator for the corresponding high-resolution visual signals, respectively. pre The optimal encoding network for high-resolution visual signals, the corresponding encoding features of high-resolution visual signals, the corresponding decoding network (i.e., the generator network), and the corresponding discrimination network for high-resolution visual signals are obtained.

[0097] Step 2: Design a haptic-assisted visual signal super-resolution reconstruction HaSR model; the structure diagram of the super-resolution reconstruction HaSR model is shown below. Figure 2 Phase 2 is shown below:

[0098] After acquiring visual and tactile signals from the terminal, the visual signals are first downsampled at the edge nodes to obtain low-resolution visual signals; then, a pre-trained, widely used coding network is used to perform preliminary feature extraction on the low-resolution visual signals and the corresponding tactile signals.

[0099] Then, by leveraging intramodal discriminativeness and intermodal consistency, a mapping network is used to reduce intermodal differences and mine the correlations between different modes to bridge the semantic gap between modes. Thus, based on the preliminary features extracted by encoding, mapping features with semantic discriminativeness and semantic association are obtained. The obtained mapping features are then normalized and passed through a channel model for use in the next step of feature fusion.

[0100] Based on the mapping features of low-resolution visual signals and tactile signals, as well as the encoding features of the obtained high-resolution visual signals, and combined with the powerful data fitting ability of generative adversarial networks, fused features are obtained.

[0101] Finally, the obtained fused features are input into the high-resolution visual signal generation network to achieve the reconstruction of the high-resolution visual signal.

[0102] The specific implementation steps are as follows:

[0103] Step 2-1: Preliminary feature extraction of low-resolution visual signals and corresponding tactile signals, based on... The paired low-resolution visual signals present in i and tactile signal h iSpecifically, for tactile signals, the DenseNet-121 network is used, and for low-resolution visual signals, the VGG-16 network is used to perform preliminary feature extraction of both low-resolution visual and tactile signals, obtaining the corresponding 512-dimensional encoded features f of the low-resolution visual signal. l and the corresponding 512-dimensional tactile signal encoding feature f t ;

[0104] Step 2-2: Encoding features f based on the obtained low-resolution visual signal l and the corresponding tactile signal encoding features f t A feature mapping network is constructed, comprising a low-resolution visual signal feature mapping network consisting of 512-256-128 dimensional fully connected layers and a tactile signal mapping network consisting of 512-256-128 dimensional fully connected layers. This effectively reduces intermodal heterogeneity differences. It learns intramodal discriminative representations and intermodal consistent representations from both intramodal and cross-modal perspectives, ultimately obtaining mapped features, including 128-dimensional low-resolution visual signal mapping features z. l Mapping features z of 128-dimensional tactile signals t Then the obtained mapping features are input into the channel model, and the corresponding feature fusion steps are executed after the main terminal receives them.

[0105] Cross-modal semantic relevance learning: Triple loss is used for cross-modal semantic relevance learning. After learning through the mapping network, the following effect is achieved: for low-resolution visual and tactile signal features from the same category, the distance between them should be close; for low-resolution visual and tactile signal features from different categories, the distance between them should be far. The loss function is defined as follows:

[0106]

[0107]

[0108] Where θ l and θ t Let p and q represent the parameters of the corresponding low-resolution visual signal mapping network and tactile signal mapping network, respectively. p and q represent different categories, N represents the number of instances of the corresponding low-resolution visual signal and tactile signal, σ represents the corresponding threshold, and L2 = ||·||2 represents the corresponding L2 norm. The total loss of semantic relevance can be expressed as the sum of the above two, i.e.:

[0109]

[0110] Discriminative learning within the same modality: While ensuring semantic relevance, it is necessary to effectively solve the semantic discrimination problem within the same modality. That is, for samples within the same modality (visual modality or tactile modality), samples belonging to the same category should be closer together, and samples belonging to different categories should be farther apart. This is mainly achieved by adding a common classifier after the mapping network. This classifier consists of fully connected layers of 128, 32, and 9 dimensions. The specific loss is expressed as follows:

[0111]

[0112] Where, p i (z) represents the probability distribution predicted by the classifier, y i It's a real label, θ c The parameters represent the corresponding common classifier; after the above processing, the mapping features z of the obtained low-resolution visual signal are... l Mapping characteristics z of tactile signals t The input channel model is normalized;

[0113] Steps 2-3: After transmission through the channel, the main terminal obtains the mapping features z of the corresponding noisy, low-resolution visual signal. l-n Mapping characteristics z of tactile signals t-n The purpose of this step is to utilize the complementarity of the two to generate fusion features, making the generated fusion features as similar as possible to the encoded features z of the high-resolution visual signal in step 1. h Similarly, based on this objective, generative adversarial networks (GANs) are chosen to perform feature fusion tasks by leveraging their ability to fit data distributions; specifically, z is first... l-n and z t-n The data is then concatenated and input into a generative network consisting of fully connected layers of 256-128 dimensions, where z... h z is considered a real sample l-n and z t-n z is considered as input to the generator m The loss of the fusion network, representing the obtained fusion features, is defined as follows:

[0114] L m =L m-adv (G m D m )+L2(z m ,z h ),

[0115] The first term represents the loss of a typical generative adversarial network, G. m Representative feature fusion network, D mThis represents the feature discrimination network corresponding to the feature fusion network. This network consists of 128, 64, and 1-dimensional fully connected layers. Specifically, this term is expressed as follows:

[0116]

[0117] Where θ gm θ represents the corresponding parameter of the fusion network generator. dm The first term represents the model parameters of the discriminant network corresponding to the fusion network generator; the second term represents the L2 loss, which helps maintain semantic consistency.

[0118] Steps 2-4: After obtaining the fusion features, the high-resolution visual signal is reconstructed using these features. The high-resolution visual signal decoding (generation) network and the corresponding discrimination network obtained in step 1 are used, and their network structure and parameters are used as the initial network parameters for this step. Furthermore, the loss L in step 1-2... pre Building upon this foundation, a perceptual loss is added to make the generated visual signal more consistent with human perceptual characteristics. The loss in this step is specifically represented as follows:

[0119] L finet =L per +βL adv-finet +γL rec ,

[0120] The first term is the perceptual loss, which can be specifically expressed as:

[0121]

[0122] Where M i,j F represents the number of parameters in the corresponding feature map. i,j The first term represents the output after the i-th convolutional layer and before the j-th max-pooling layer of the VGG-19 network, which we set here as i=4, j=5. The second term is the loss of the network generating the high-resolution visual signal, which can be specifically expressed as...

[0123]

[0124] Where θ gh This represents the corresponding utilization of fusion feature z m The parameters of the generative network used for high-resolution visual signal reconstruction, θ dh The first term represents the parameters of the corresponding discrimination network, the third term is the reconstruction loss; in addition, β and γ are hyperparameters.

[0125] Step 3: Train the HaSR model using optimization methods to obtain the optimal model parameters for subsequent testing.

[0126] Step 3-1: Use the existing high-resolution visual signals to train the encoding network and the corresponding decoding (generation) network of the high-resolution visual signals, as well as the corresponding discrimination network of the high-resolution visual signals. The specific process is as follows:

[0127] Step 311: Initialize the parameter θ eh (0), θ gh (0), θ dh (0) is the value of the corresponding parameter in the 0th iteration;

[0128] Step 312: Set the number of iterations to n1 = 3000 and the learning rate to μ1 = 0.0008;

[0129] Step 313: Use the RMSProp algorithm to optimize the network parameters:

[0130]

[0131]

[0132]

[0133] where θ eh (n + 1), θ dh (n + 1), θ gh (n + 1) and θ eh (n), θ dh (n), θ gh (n) are the parameters of the encoding network of the high-resolution visual signals, the discrimination network of the high-resolution visual signals, and the generation network of the high-resolution visual signals corresponding to the (n + 1)th and nth times respectively, is to take the partial derivative of each loss function;

[0134] Step 314: If n < n1, repeat Step 313. After n1 rounds of iteration, obtain the network that converges to the optimal, including the encoding network of the high-resolution visual signals, the discrimination network of the high-resolution visual signals, and the generation network of the high-resolution visual signals;

[0135] Step 3-2: Based on the encoding network of the high-resolution visual signals, the decoding (generation) network of the high-resolution visual signals, and the corresponding discrimination network obtained in the first step, use the low-resolution visual signals and the corresponding tactile signals to train the encoding and mapping network of the corresponding low-resolution visual signals, the encoding and mapping network of the tactile signals, and the feature fusion network, and fine-tune the generation network of the corresponding high-resolution visual signals. The encoding network of the low-resolution visual signals and the encoding network of the tactile signals are obtained by loading the pre-trained network and do not participate in the update in this step. The specific process is as follows:

[0136] Step 321: Initialize the parameter θl (0), θ t (0), θ c (0), which represents the initial random parameters of the corresponding network. The parameters of the generator and discriminator networks that obtained the high-resolution visual signal in step 3-1 are loaded as θ. gh (0), θ dh (0) initial parameters;

[0137] Step 322: Start the iteration, set the number of iterations to n2 = 2000, and the learning rate to μ2 = 0.0015;

[0138] Step 323: Use the Adam algorithm to optimize the parameters of the low-resolution visual signal mapping network, the tactile signal mapping network, and the common classifier:

[0139]

[0140]

[0141]

[0142] Where θ c (n+1), θ l (n+1), θ t (n+1) and θ c (n), θ l (n), θ t (n) represents the parameters of the common classifier for the (n+1)th and nth iterations, respectively, the low-resolution visual signal mapping network, and the corresponding tactile signal mapping network. To take partial derivatives with respect to each loss function;

[0143] Step 324: Use the Adam algorithm to optimize the parameters of the feature fusion network and the corresponding discriminant network for the fused features:

[0144]

[0145]

[0146] Where θ gm (n+1), θ dm (n+1), θ gm (n) and θ dm (n) are the parameters of the generation network and the discrimination network for the fused features corresponding to the (n+1)th and nth iterations, respectively. To take partial derivatives with respect to each loss function;

[0147] Step 325: Using the RMSProp algorithm, fine-tune the generation and discrimination networks for the high-resolution visual signals, and optimize and update them using the following function:

[0148]

[0149]

[0150] where θ gh (n + 1), θ dh (n + 1), θ gh (n) and θ dh (n) represent the parameters corresponding to the generation network and the discrimination network of the high - resolution visual signals corresponding to the (n + 1)-th and the n-th respectively. is to take the partial derivative of each loss function;

[0151] Step 326: If n < n2, then jump to Step 323. After n2 rounds of iteration, an HaSR network converging to the optimal is obtained, including a mapping network for low - resolution visual signals, a mapping network for tactile signals, a feature fusion network, and a discrimination network for the corresponding fused features, as well as a discrimination network for high - resolution visual signals and a generation network for high - resolution visual signals that converge to the optimal after fine - tuning.

[0152] Step 4: Input the paired low - resolution visual signals and tactile signals to be measured into the optimal HaSR model. The optimal HaSR model is used to extract the features of the low - resolution visual signals and the corresponding tactile signals and fuse them, and generate the required high - resolution visual signals using the fused features.

[0153] Step 4 - 1: Adopt the trained HaSR model;

[0154] Step 4 - 2: Input a set of paired low - resolution visual signals and corresponding tactile signals into the HaSR model, complete the encoding, mapping, and fusion of modal features, and finally obtain the corresponding high - resolution visual signals.

[0155] The following experimental results show that, compared with the existing methods, the present invention realizes the super - resolution reconstruction of visual signals by using the consistency and complementarity of multi - modal signals and achieves better generation effects.

[0156] This invention utilizes the LMT-108 multimodal dataset for experiments, which is commonly used for cross-modal retrieval and generation tasks. It contains 108 materials commonly found in daily life. These different surface materials can be broadly categorized into nine classes based on their physical properties: mesh, stone, metal, wood, rubber, fiber, foam, foil and paper, textiles, and fabrics. Each class further comprises 5-17 subclasses. For each class, the dataset includes visual signals of various texture types and triaxial (X, Y, Z) acceleration signals generated by sliding or tapping. Based on previous work using this dataset for cross-modal learning, the visual signal samples in this invention are primarily represented by non-flickering RGB visual signals; for tactile data, the z-axis acceleration signal exhibits the most significant vibration during the movement of the sensor, therefore the z-axis acceleration signal is selected as the tactile signal. The original high-resolution visual signal is represented as 128×128. The low-resolution visual signal is obtained by downsampling by 4×, so its size is 32×32. For the tactile signal, considering the pressure effect when the sensor touches and leaves, only the middle tactile signal is extracted. Then we obtain the corresponding spectrum through STFT transformation.

[0157] Existing Method 1: The PIX2PIX method in the literature "Image-to-image translation with conditional adversarial networks" (authors P. Isola, J.-Y. Zhu, T. Zhou, and AAEfros) is a classic method that applies generative adversarial networks (GANs) to supervised translation of visual signals of different styles. This invention uses the spectrogram of tactile signals and paired high-resolution visual signals as paired training data, and uses the spectrogram of tactile signals as conditional information to generate high-resolution visual signals.

[0158] Existing Method Two: The Discogan method in the paper "Learning to discover cross-domain relations with generative adversarial networks" (authors T. Kim, M. Cha, H. Kim, J. K. Lee, and J. Kim) utilizes GANs to discover relationships between different domains and achieve the conversion from one domain to another. This invention achieves the conversion from the spectrogram of tactile signals to high-resolution visual signals.

[0159] Method 3: Bilinear interpolation is a commonly used method for upsampling visual signals, directly interpolating low-resolution visual signals to obtain clearer visual signals. It has a smoothing function and can effectively overcome the shortcomings of traditional neighborhood interpolation.

[0160] Existing Method 4: The paper "Photo-realistic single image super-resolution using agency adversarial network" (authors C. Ledig, L. Theis and F. Huszar et al.) proposed SRGAN, which was the first method to introduce GAN into the field of super-resolution reconstruction, and obtained visual signals that are more consistent with human perception by introducing perceptual loss.

[0161] Existing Method 5: The paper "Esrgan: Enhanced super-resolution generative adversarial networks" (authors X. Wang, K. Yu, S. Wu et al.) introduces a densely connected residual block on the basis of SRGAN to achieve deeper network training, and further improves visual quality by balancing perceptual quality and fidelity.

[0162] This invention: The method of this embodiment.

[0163] In this embodiment, the performance indicators used to evaluate the super-resolution reconstruction scheme proposed in this invention are divided into three categories: peak signal-to-noise ratio, structural similarity, and Frechet Inception distance.

[0164] Peak Signal-to-Noise Ratio (PSNR): PSNR is a visual signal quality evaluation metric. Based on the error between pixels, it calculates the ratio of the energy of the peak signal to the average energy of the noise. It is the most common and widely used objective evaluation metric for visual signals. A higher PSNR value indicates less distortion.

[0165] Structural similarity: Structural similarity (SSIM) measures the similarity of visual signals from three aspects: brightness, contrast, and structure. The value of SSIM ranges from [0,1]. The larger the value, the smaller the distortion of the visual signal. Structural similarity is more consistent with human perception than PSNR in assessing the quality of visual signals.

[0166] Frechet Inception Distance (FID): The Frechet Inception distance (FID) is used to evaluate the similarity between visual signals generated by generative adversarial networks and real visual signals. It calculates the distance between real and generated visual signals in the feature space. First, an Inception network is used to extract features. Then, a Gaussian model is used to model the feature space, and finally, the distance between two features is calculated. A lower FID indicates higher image quality and diversity.

[0167] Table 1. Performance Comparison Results of Super-Resolution Reconstruction Schemes

[0168]

[0169]

[0170] From Table 1 and Figure 3 As can be seen, compared with the aforementioned competitive methods, the method proposed in this invention has significant advantages. Among the five comparative schemes, the super-resolution reconstruction method based on cross-modal fusion exhibits better performance. This result demonstrates that the intra-modal and inter-modal mapping feature learning proposed in this invention can better uncover modal properties, and the effective feature fusion method can utilize the complementarity between different modalities. Finally, it achieves improved reconstruction quality of visual signals using a reconstruction network, making them more similar to the original visual signals.

[0171] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. An image super-resolution reconstruction method for cross-modal communication, characterized in that: The image super-resolution reconstruction method comprises the following steps: Step 1, using complete high-resolution visual signals, encoding and decoding of high-resolution visual signals, training the encoding network of high-resolution visual signals through the encoding step, and obtaining the encoding features of high-resolution visual signals, training the decoding network of high-resolution visual signals through the decoding step, and providing support for the subsequent visual signal super-resolution reconstruction model; Step 2, design a haptic-assisted visual signal super-resolution reconstruction HaSR model; the super-resolution reconstruction HaSR model is: After collecting visual signals and haptic signals from the terminal, down-sampling the visual signals at the edge node to obtain low-resolution visual signals, and using the pre-trained widely used encoding network to preliminarily extract features of the low-resolution visual signals and the corresponding haptic signals; Using the intra-modal discriminability and inter-modal consistency, the mapping network is used to reduce the difference between the modes, and the correlation of different modes is mined to make up for the semantic gap between the modes, so as to obtain mapping features with semantic discriminability and semantic correlation based on the preliminary features extracted by encoding, and then the mapping features are normalized and passed through the channel model, so as to be used for feature fusion in the next step; According to the mapping features of the low-resolution visual signals and the mapping features of the haptic signals and the obtained encoding features of the high-resolution visual signals, the powerful data fitting capability of the generative adversarial network is used to obtain the fusion features; Finally, the obtained fusion features are input into the generative network of high-resolution visual signals to realize the reconstruction of high-resolution visual signals; Step 3, using a model optimization algorithm to train the HaSR model designed in step 2, and finally obtaining the optimal model parameters for the test stage; Step 4, input the to-be-tested pair of low-resolution visual signals and haptic signals into the optimal HaSR model, and the optimal HaSR model is used to extract the features of the low-resolution visual signals and the corresponding haptic signals and perform fusion, and the high-resolution visual signals required are generated by using the fused features, Wherein, step 1 specifically comprises the following steps: Step 1-1: For training dataset which contains paired haptic, low-resolution visual signal and high-resolution visual signal, N is the number of paired visual signal and haptic signal, d i = {h i , l i , t i} respectively represent high-resolution visual signal, low-resolution visual signal and corresponding haptic signal, the i-th high-resolution visual signal h i is input into the encoding network of high-resolution visual signal, and the encoding feature z h of the visual signal is extracted; Step 1-2: encode the obtained high-resolution visual signal into a feature z h The input is input into the decoder of the high-resolution visual signal composed of the generative adversarial network, and then the decoder reconstructed high-resolution visual signal is input into the discriminator of the high-resolution visual signal, the encoder and the decoder are trained jointly, and the reconstruction loss and the discrimination loss are used to optimize, finally the encoding feature of the high-resolution visual signal and the corresponding decoding network are learned, and the defined loss function is: L pre = L rec + aL pre-adv Wherein, α is a coefficient for adjusting the proportion of different losses, and the first loss is a reconstruction loss: where (C, H, W) is the size of the high-resolution visual signal, G h a decoding network representing the corresponding high-resolution visual signal, an encoding feature of the high-resolution visual signal obtained by the corresponding encoding network, ||·||1 represents the corresponding L1 norm, and the second term is the loss of the generative adversarial network, and the specific loss function is: where E(*) denotes the expectation value of the distribution function, p(z h ) represents the distribution of the high-resolution image encoding features, p(h) represents the distribution of the real high-resolution image, D h represents the corresponding high-resolution visual signal discriminator network, used to complete the judgment of the reconstructed high-resolution visual signal, θ gh and θ dh represent the parameters of the corresponding high-resolution visual signal generator and discriminator, respectively, and the optimal high-resolution visual signal encoding network and the corresponding high-resolution visual signal encoding features, the corresponding decoding network and the corresponding high-resolution visual signal discriminator network are obtained by minimizing L pre .

2. The image super-resolution reconstruction method for cross-modal communication according to claim 1, characterized in that: Step 2 comprises the following steps: Step 2-1: Preliminary feature extraction of low-resolution visual signals and corresponding haptic signals based on training dataset The paired low-resolution visual signals l i and haptic signals h i present in the middle are preliminarily extracted by using a deep neural network to obtain the corresponding encoded features f l of low-resolution visual signals and haptic signals t ; Step 2-2: encoding features f based on the obtained low-resolution visual signal l and the corresponding haptic signal encoding features f t , a feature mapping network is established to effectively reduce the heterogeneity difference between modalities, learn the intra-modal discriminative representation and the inter-modal consistent representation from the perspectives of intra-modality and cross-modality, and finally obtain the mapping features, which include the mapping features z l of the low-resolution visual signal and the mapping features z t of the haptic signal, and then input the obtained mapping features into the channel model and perform the corresponding feature fusion step after being received by the primary terminal; Cross-modal semantic correlation learning: a triplet loss is selected to learn the cross-modal semantic correlation, and the mapping network is learned to achieve the following effects, that is, the distance between the low-resolution visual signal features and the haptic signal features from the same category should be close, and the distance between the low-resolution visual signal features and the haptic signal features from different categories should be far away; the loss function is defined as follows: where θ l and θ t represent the parameters of the corresponding low-resolution visual signal mapping network and the mapping network of the haptic signal, p and q represent different categories, N represents the number of instances of the corresponding low-resolution visual signal and haptic signal, σ represents the corresponding threshold value, L2=||·||2 represents the corresponding L2 norm, and the total loss of semantic correlation is represented as the sum of the above two, i.e. Intra-modal discriminability learning: while ensuring semantic correlation, effectively solve the intra-modal semantic discriminability problem, that is, the samples in the same mode belong to the same category and the distance is closer, and the samples belong to different categories and the distance is farther, which is completed by adding a public classifier after the mapping network, and the specific loss is represented as follows: where p i (z) represents the probability distribution predicted by the classifier, y i is the true label, θ c represents the parameters of the corresponding public classifier; after the above processing, the obtained mapping features of the low-resolution visual signal z l and the mapping features of the tactile signal z t are normalized and input into the channel model; Step 2-3: After transmission through the channel, the main terminal obtains the corresponding noisy low-resolution visual signal mapping feature z l-n and the mapping feature z of the tactile signal t-n , using the ability of the generative adversarial network to fit the data distribution to complete the feature fusion task, wherein z h is regarded as a real sample, z l-n and z t-n are regarded as the input of the generator, and z m represents the obtained fusion feature, and the loss of the defined fusion network is as follows: L m = L m-adv (G m , D m ) + L2(z m , z h ), wherein the first term represents the loss of the ordinary generative adversarial network, G m represents the feature fusion network, D m represents the feature discrimination network corresponding to the feature fusion network, and the term is specifically represented as: where E (*) denotes the expectation value of the distribution function, p(z h ) denotes the distribution of the high-resolution image encoding features, p(z m ) represents the distribution of the fused features, θ gm represents the corresponding parameters of the fusion network generator, and θ dm represents the model parameters of the discriminator network corresponding to the fusion network generator, and the second term represents the L2 loss, which is beneficial to maintaining semantic consistency; Step 2-4: After obtaining the fusion features, high-resolution visual signal reconstruction is realized by using the fusion features. The high-resolution visual signal decoding network obtained in step 1 and the corresponding discrimination network are used as the initialization network parameters of this step, and the network structure and parameters are used as the initialization network parameters of this step. On the basis of the loss L pre in step 1-2, the perceptual loss is added to make the generated visual signal more consistent with the human perception characteristics. The loss is specifically represented as follows: L finet = L per + βL adv-finet + γL rec , Wherein, the first item is a perception loss, which is specifically represented as: where M i,j represents the number of parameters of the corresponding feature map, F i,j represents the output before the jth max-pooling layer after the ith convolutional layer of the VGG-19 network, β and γ are hyperparameters, and the second term is the loss of the generation network of the high-resolution visual signal, which is specifically represented as where θ gh represent the corresponding fusion feature z m parameters of the generation network for high-resolution visual signal reconstruction, θ dh represent the corresponding parameters of the discriminator network, and the third term is the reconstruction loss.

3. The image super-resolution reconstruction method for cross-modal communication according to claim 1, characterized in that: Step 3 specifically comprises the following steps: Step 3-1: training the encoding network and the corresponding decoding network of the high-resolution visual signal, and the corresponding discrimination network of the high-resolution visual signal, using the existing high-resolution visual signal; Step 3-2: based on the encoding network of the high-resolution visual signal, the decoding network of the high-resolution visual signal and the corresponding discrimination network obtained in step 3-1, using the low-resolution visual signal and the corresponding tactile signal, training the corresponding encoding and mapping network of the low-resolution visual signal, the encoding and mapping network of the tactile signal, and the feature fusion network, and fine-tuning the corresponding generation network of the high-resolution visual signal, wherein the low-resolution visual signal encoding network and the tactile signal encoding network are obtained by loading the pre-trained network and are not updated in this step.

4. The image super-resolution reconstruction method for cross-modal communication according to claim 3, characterized in that: The specific process of step 3-1 is as follows: Step 311, initialize parameter θ eh (0), θ gh (0), θ dh (0) is the value of the corresponding parameter at the 0th iteration; Step 312: set the number of iterations to n1 and the learning rate to μ1; Step 313: use the RMSProp algorithm to optimize the network parameters: where θ eh (n + 1), θ dh (n + 1), θ gh (n + 1) and θ eh (n), θ dh (n), θ gh (n) are parameters of the encoding network, the discriminator network, and the generation network of the high-resolution visual signal, respectively, at the n+1th and nth iteration, is the partial derivative of each loss function. Step 314: if n < n1, repeat step 313, and after n1 iterations, obtain the network converged to the optimal, including the encoding network of the high-resolution visual signal, the discrimination network of the high-resolution visual signal, and the generation network of the high-resolution visual signal.

5. The image super-resolution reconstruction method for cross-modal communication according to claim 3, characterized in that: The specific process of step 3-2 is as follows: Step 321, initialize parameter θ l (0), θ t (0), θ c (0), which represents the initial random parameter of the corresponding network, load the parameters of the generation network and the discrimination network obtained in step 3-1 to obtain the high-resolution visual signal as θ gh (0), θ dh (0) initial parameters; Step 322: start iteration, set the number of iterations to n2 and the learning rate to μ2; Step 323: use the Adam algorithm to optimize the parameters of the low-resolution visual signal mapping network and the tactile signal mapping network and the public classifier: where θ c (n+1), θ l (n+1), θ t (n+1) and θ c (n), θ l (n), θ t (n) are the parameters of the corresponding public classifier at the n+1th and nth time, respectively, the low-resolution visual signal mapping network and the corresponding haptic signal mapping network, is the partial derivative of each loss function. Step 324: use the Adam algorithm to optimize the parameters of the feature fusion network and the corresponding discrimination network of the fused features: where θ gm (n+1), θ dm (n+1), θ gm (n) and θ dm (n) are the parameters of the generation network and the discrimination network of the fused feature corresponding to the (n+1)th and nth time, respectively, is the partial derivative of each loss function. Step 325: use the RMSProp algorithm to fine-tune the generation network and the discrimination network of the high-resolution visual signal, and optimize and update through the following function: where θ gh (n+1), θ dh (n+1), θ gh (n) and θ dh (n) represent the parameters of the generation network and the discriminator network corresponding to the (n+1)th and the nth high-resolution visual signal, respectively, is the partial derivative of each loss function. Step 326: if n < n2, jump to step 323, and after n2 iterations, obtain the HaSR network converged to the optimal, including the mapping network of the low-resolution visual signal, the mapping network of the tactile signal, the feature fusion network and the corresponding discrimination network of the fused features, and the fine-tuned high-resolution visual signal discrimination network and the generation network of the high-resolution visual signal converged to the optimal.

6. The image super-resolution reconstruction method for cross-modal communication according to claim 1, characterized in that: Step 4 includes the following steps: Step 4-1: use the trained HaSR model; Step 4-2: input a set of paired low-resolution visual signals and corresponding tactile signals into the HaSR model to complete the encoding, mapping and fusion of the modal features, and finally obtain the corresponding high-resolution visual signal.

Images