Signal-to-noise ratio adaptive joint optimization information source channel tactile signal reconstruction method

Abstract

The invention discloses a tactile signal reconstruction method of a signal-to-noise ratio self-adaptive joint optimization information source channel, which relates to the technical field of tactile signal generation, and comprises the following steps: preprocessing an image corresponding to each object and a tactile vibration signal generated by sliding; constructing a cross-modal joint encoder for generating hierarchical fusion features, constructing a cross-modal joint decoder, and inputting training data obtained by preprocessing into the constructed cross-modal joint encoder and the cross-modal joint decoder for training; and inputting the to-be-detected paired tactile signal and image signal into the cross-modal joint encoder and the optimal cross-modal joint decoder, and reconstructing a target tactile signal. According to the tactile signal reconstruction method with the signal-to-noise ratio adaptive joint optimization information source channel provided by the invention, the coding training process is simplified, and modal feature fusion is better carried out; the robustness of the model to channel change is improved, so that the model shows higher tactile signal reconstruction stability under a larger range of signal-to-noise ratio change.

Description

Technical Field

[0001] The present invention relates to the technical field of tactile signal generation, and in particular to a tactile signal reconstruction method for signal-to-noise ratio adaptive joint optimization of a source channel. Background Art

[0002] With the advancement of science and technology and the continuous evolution of user needs, traditional audio-visual media are no longer sufficient to meet users' growing expectations for immersive and interactive experiences, which has promoted the evolution of multimedia services to a more advanced form. However, existing research has mostly focused on the analysis and transmission of single-modal signals, and has failed to fully address the complex transmission requirements of heterogeneous multimodal signals. Therefore, cross-modal services have emerged, which have created a new communication mode and experience for users by integrating multiple sensory information such as vision, hearing, and touch. In particular, cross-modal services based on touch are widely considered to be the key development direction of future communication technologies because they can provide direct and real interactive feedback. Despite this, the current research on cross-modal services centered on touch still faces several challenges.

[0003] On the one hand, the current cross-modal signal reconstruction schemes have not been fully studied on the impact of channel noise, and the robustness of signal reconstruction under different channel conditions needs to be considered more deeply. In actual communication scenarios, tactile signals need to be encoded at the source end, transmitted in the channel, and decoded and reconstructed at the receiving end. This process involves the interaction of multiple factors, including the semantic fusion of cross-modal signals, the transmission characteristics of the channel, and the reconstruction strategy at the receiving end.

[0004] On the other hand, the existing signal communication reconstruction model has low generalization of signal-to-noise ratio in practical applications, and requires network training and deployment according to different channel conditions, which takes up more computing and storage resources. Existing models based on deep neural networks rely on data-driven strategies. The optimization process of the entire system is based on an end-to-end loss function of recovery accuracy, and the parameters of the model are adjusted through supervised learning. This model training mechanism easily leads to the model's low generalization of signal-to-noise ratio, which is manifested in the model's good performance under certain specific signal-to-noise ratios during training, while the performance drops rapidly under other signal-to-noise ratios.

[0005] Existing research has also made some progress in the cross-modal fusion and generation of tactile signals. In the paper “Information Recovery Technology for Cross-Modal Communications”, Xu et al. proposed an information recovery scheme to address the problem that multimodal data (such as audio, video, and tactile signals) may be lost or contaminated by wireless channel noise during transmission. The scheme uses existing multimodal data at the receiving end to achieve information recovery through one-to-one retrieval of the same modality, one-to-one retrieval of cross-modality, and one-to-many retrieval of cross-modality. In the paper “Haptic Signal Reconstruction for Cross-Modal Communications”, Wei et al. proposed a method for audio-visual assisted tactile signal reconstruction based on cloud-edge collaboration. In the case of insufficient data received by edge nodes, a large audio-visual auxiliary database stored in the central cloud is used to acquire and transmit knowledge and migrate it to edge nodes. Then, the edge nodes are used to mine the inherent semantic consistency and correlation between modalities, and finally the desired tactile signal reconstruction is achieved. The tactile signal reconstruction is achieved by considering the received audio-visual signal and associating it with the potential correlation of the tactile modality.

[0006] However, most of these models are limited to one-way conversion at the transmitter or receiver, without considering the end-to-end design of the entire communication process. They also lack simulation of transmission conditions in actual channels. In actual situations, they are easily interfered by noise, resulting in reduced data quality after transmission. This will have an adverse impact on multimodal data streams that require low latency and high reliability, especially tactile streams. In addition, how to design effective channel adaptation technology to significantly improve the stability of tactile signal reconstruction in response to diverse channel conditions and variable channel signal-to-noise ratios is also a difficult problem faced by tactile-based cross-modal communications. Summary of the invention

[0007] In view of the above-mentioned problems, the present invention is proposed.

[0008] Therefore, the present invention implements the signal encoding, transmission and decoding process based on deep neural network. On the one hand, the unified model design of cross-modal joint codec simplifies the encoding training process.

[0009] At the same time, the unified model also brings a unified tactile-image signal feature representation, which can better perform modal feature fusion; on the other hand, through the separately designed signal-to-noise ratio adaptive module, the model's robustness to channel changes is improved, so that the model can show higher tactile signal reconstruction stability under a wider range of signal-to-noise ratio changes.

[0010] In order to solve the above technical problems, the present invention provides the following technical solutions: a tactile signal reconstruction method for signal-to-noise ratio adaptive joint optimization of source channels, comprising:

[0011] Preprocessing the image signal corresponding to each object and the tactile vibration signal generated by sliding based on the tactile-image database;

[0012] The preprocessed image and tactile signal are input into the cross-modal joint encoder, the input data is processed in blocks, and then input into the hierarchical feature extraction module and the signal-to-noise ratio adaptive module in sequence to obtain the hierarchical fusion features;

[0013] After being transmitted through the channel, the hierarchical fusion features enter the cross-modal joint decoder, which includes a hierarchical feature extraction module, a signal-to-noise ratio adaptation module, and a signal reconstruction module;

[0014] The hierarchical feature extraction module decodes the low-dimensional tactile features from the hierarchical fusion features. The signal-to-noise ratio adaptive module dynamically adjusts the low-dimensional features according to the channel changes. The signal reconstruction module converts the low-dimensional features into a tactile spectrum. The tactile spectrum is converted into a tactile sequence through an inverse short-time Fourier transform (ISTFT), thereby completing the tactile signal reconstruction.

[0015] The cross-modal joint encoder and the cross-modal joint decoder are jointly trained according to the training data to obtain the optimal model parameters;

[0016] The tactile and image signals to be tested are input into the optimal cross-modal joint encoder to obtain hierarchical fusion features. After channel transmission, the received hierarchical fusion features are input into the cross-modal joint decoder to generate the target tactile spectrum, and the target tactile signal is obtained through ISTFT transformation.

[0017] As a preferred solution of the tactile signal reconstruction method of the signal-to-noise ratio adaptive joint optimization source channel of the present invention, the preprocessing includes cutting the image into H×W size, which is expressed as:

[0018]

[0019] Among them, v j Represents the jth RGB three-channel image of size H×W in the dataset;

[0020] The tactile signal is preprocessed, and the tactile sequence is subjected to short-time Fourier transform (STFT) to obtain the spectrum of the tactile sequence, which is expressed as:

[0021]

[0022] Pass the spectrum through and

[0023] , and get the amplitude spectrum and phase spectrum

[0024] The tactile signal is obtained by splicing the amplitude spectrum and the phase spectrum.

[0025] Convert RGB image and tactile signals Splice into five-dimensional data in the channel dimension Wherein, n=H×W×5 represents the size of the image tactile signal pair, and during the processing, the one-to-one pairing of the image and the tactile signal is maintained.

[0026] As a preferred solution of the tactile signal reconstruction method of the signal-to-noise ratio adaptive joint optimization source channel of the present invention, wherein: the block processing includes dividing the data pair into non-overlapping blocks of 4×4 size;

[0027] The feature dimension of each patch is 4×4×5=80, and the size of the data tensor is

[0028] The divided data is input into multiple hierarchical feature extraction modules and signal-to-noise ratio adaptive modules;

[0029] The hierarchical feature extraction module is FE e (·), the signal-to-noise ratio adaptive module is SD e (·), connected after the first and last feature extraction components, respectively;

[0030] Assume that the number of feature extraction components in the encoder is L, and the input of the jth feature extraction component is t j ,j∈{1,2,...,L}, the continuous calculation process is expressed as,

[0031]

[0032] The tactile-image signal pair d∈D is divided into non-overlapping small blocks in both length and width to obtain t1, which is input into the hierarchical feature extraction module to obtain the first layer feature representation. The first layer of feature representation is input into the signal-to-noise ratio adaptive module to obtain t2, t2 is input into multiple feature extraction components, deep features are obtained after passing through the last layer of feature extraction components, and input into the last layer of signal-to-noise ratio adaptive layer to obtain the remapped feature e2.

[0033] As a preferred solution of the tactile signal reconstruction method of the signal-to-noise ratio adaptive joint optimization source channel of the present invention, wherein: the hierarchical feature extraction module FE e (·) Includes merging layers and Swin Transformer modules to learn hierarchical features of tactile-image signals;

[0034] The merging layer is responsible for downsampling and dimension upscaling to produce features at different levels, while the SwinTransformer block performs hierarchical feature learning;

[0035] The first merging layer connects the features of each group of 2×2 adjacent patches in the channel dimension, downsampling the feature resolution by 2 times and increasing the feature dimension by 4 times;

[0036] Applying a fully connected layer on the concatenated features unifies the output dimension to twice the original dimension, i.e. 2C, and the tensor size is

[0037] After multiple passes through the Swin Transformer blocks and merging layers, a hierarchical feature representation is obtained;

[0038] The last merging layer of the cross-modal joint encoder is responsible for mapping the deep feature information to the same dimension as the channel channel size. After connecting the adjacent patch features, a linear layer is applied to transform the output dimension to the channel dimension b. At this time, the tensor size is

[0039] As a preferred solution of the tactile signal reconstruction method of the signal-to-noise ratio adaptive joint optimization source channel of the present invention, wherein: the signal-to-noise ratio adaptive module SD(·) includes a global pooling and a feedforward network;

[0040] The global pooling is used to capture the full-size information of the input features, aggregate the local features into global features, connect the signal-to-noise ratio information with the pooled features, and input the feedforward network to generate a scaling factor. The scaling factor is combined with the hierarchical feature extraction module FE e (·) to obtain the weighted features;

[0041] The output of the feature extraction component is o={o1,o2,...,o c},o ipq Indicates the element whose i-th channel is located at (p,q);

[0042] After global average pooling P(·), each feature map is pooled in the channel dimension to extract global information

[0043] The global pooling operation transforms each two-dimensional feature map o on the channel i Compressed into a single value s i , represents the average activation intensity of the entire feature map, expressed as,

[0044]

[0045] Concatenate the global information with the current SNR value to get Input it into the feedforward network, expressed as,

[0046]

[0047] Will Compared with the original input features before pooling Multiply to get the features adjusted according to the channel conditions Since the two dimensions are different, the data of each channel of factor is copied before multiplication, thus expanding the dimension to H×W×c.

[0048] As a preferred solution of the tactile signal reconstruction method of the signal-to-noise ratio adaptive joint optimization source channel described in the present invention, wherein: the cross-modal joint decoder uses an extension layer to upsample the extracted deep features, the extension layer reshapes the feature map of adjacent dimensions, and reduces the feature dimension to half of the original dimension, thereby completing the dimensionality reduction of the feature;

[0049] The channel signal is input into multiple hierarchical feature extraction modules to obtain features at different levels. The signal-to-noise ratio adaptive module is connected after the first and last hierarchical feature extraction modules;

[0050] Assume that the number of hierarchical feature extraction modules in the cross-modal joint decoder is L, and the input to the jth hierarchical feature extraction module according to the data flow is t j ,j∈{1,2,...,L}, the continuous calculation process is expressed as,

[0051]

[0052]

[0053] The signal transmitted from the channel Input the hierarchical feature extraction module to obtain the first layer of dimensionality reduction feature representation The input signal-to-noise ratio adaptive module obtains t2. After t2 is reduced in dimension by multiple hierarchical feature extraction modules, it passes through a signal-to-noise ratio adaptive layer again to perform feature scaling on the shallow features after the final dimension reduction to obtain the remapped feature e2. It is scaled 4 times by an expansion layer to convert the feature size into

[0054] The feature input signal reconstruction module after dimensionality reduction is mapped from the feature domain to the time-frequency matrix containing the amplitude and phase information of the tactile sequence through multiple convolutional layers, which is expressed as,

[0055]

[0056] The multi-level features are input into multiple convolutions to keep the size unchanged and achieve the scaling of the number of channels. The number of channels is scaled to 2 and mapped to the time-frequency matrix of the amplitude and phase information of the tactile sequence. The restored tactile sequence is obtained through ISTFT transformation.

[0057] As a preferred solution of the tactile signal reconstruction method of the signal-to-noise ratio adaptive joint optimization source channel of the present invention, wherein: the joint training includes selecting Charbonnier loss as the loss function of the optimization process, expressed as,

[0058]

[0059] Among them, h j represents the tactile signal recovered by the decoder, h j Represents the real tactile signal, ε=10 -3 is a constant;

[0060] The cross-modal joint encoder and the cross-modal joint decoder are jointly trained as a whole network, the gradient is calculated by the Adam optimization algorithm, and the network parameters are updated by back propagation. The expression is as follows:

[0061]

[0062] Among them, m t and v t are unbiased estimates of the first and second order moments, respectively. t is the gradient of the parameter, beta1 and beta2 are the decay rates of the first-order moment estimate and the second-order moment estimate, respectively. μ is the learning rate, and ε is a small constant used to ensure that the denominator is not zero. θ t and θ t+1 They are the network parameters before and after the update, respectively. The network parameters here refer to the network parameters composed of the cross-modal joint encoder and the cross-modal joint decoder. The learning rate is selected as 1×10-4 and the number of iterations is 1000. After the iteration, the network parameters are determined and the model training is completed.

[0063] As a preferred solution of the tactile signal reconstruction method of the signal-to-noise ratio adaptive joint optimization source channel described in the present invention, the reconstructed target tactile signal includes passing through a cross-modal joint encoder to generate a hierarchical fusion feature, which is then transmitted through the channel and decoded by a cross-modal joint decoder to reconstruct the spectrum of the target tactile signal, and the desired tactile signal is obtained through ISTFT transformation.

[0064] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the tactile signal reconstruction method of adaptively jointly optimizing a source channel with a signal-to-noise ratio as described above are implemented.

[0065] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for tactile signal reconstruction by adaptively joint optimizing a source channel with a signal-to-noise ratio as described above.

[0066] Beneficial effects of the present invention: The tactile signal reconstruction method of the signal-to-noise ratio adaptive joint optimization source channel provided by the present invention gives full play to the advantages of multimodal feature fusion and realizes the effective fusion of semantic information of different modalities. It can not only significantly compress the source signal and reduce the amount of data, but also maintain the integrity and accuracy of the information. Furthermore, in order to cope with the impact of channel noise and changes on signal transmission, the present invention specially designs a set of channel codecs. The codec can improve the robustness of the signal to channel changes and ensure that the signal can remain stable and reliable under various channel conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0068] Figure 1 An overall flow chart of a tactile signal reconstruction method for adaptively and jointly optimizing a source channel of a signal-to-noise ratio is provided in accordance with an embodiment of the present invention.

[0069] Figure 2 A schematic diagram of the network structure of a tactile signal reconstruction method for adaptively jointly optimizing a source channel of a signal-to-noise ratio provided by an embodiment of the present invention.

[0070] Figure 3 A schematic diagram of a Swin Transformer module of a tactile signal reconstruction method for adaptively jointly optimizing a source channel with a signal-to-noise ratio provided by an embodiment of the present invention.

[0071] Figure 4 A schematic diagram of a signal-to-noise ratio adaptation module of a tactile signal reconstruction method for a signal-to-noise ratio adaptively joint optimized source channel provided by an embodiment of the present invention.

[0072] Figure 5 A diagram showing the tactile signal reconstruction results of a tactile signal reconstruction method for a signal-to-noise ratio adaptively joint optimized source channel provided by an embodiment of the present invention and other comparative methods.

[0073] Figure 6A diagram of tactile spectrum and sequence reconstruction results of a tactile signal reconstruction method for a signal-to-noise ratio adaptive joint optimization source channel provided by an embodiment of the present invention.

[0074] Figure 7 A comparison chart of tactile reconstruction stability of a tactile signal reconstruction method for adaptively jointly optimizing a source channel using a signal-to-noise ratio provided by an embodiment of the present invention and other methods. DETAILED DESCRIPTION

[0075] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0076] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0077] Example 1

[0078] Reference Figure 1-Figure 4 , which is an embodiment of the present invention, provides a tactile signal reconstruction method for signal-to-noise ratio adaptive joint optimization of source channels, comprising:

[0079] Step 1: Preprocess the image signal corresponding to each object and the tactile vibration signal generated by sliding based on the public tactile-image database. The tactile vibration signal generated by sliding is converted into amplitude spectrum and phase spectrum through short-time Fourier transform (STFT), and the image signal is resized to align the tactile spectrum size;

[0080] Step 1-1: First, pre-process the image signal. Crop the image to H×W size, expressed as: Among them, v j Represents the jth RGB three-channel image of size H×W in the dataset.

[0081] Step 1-2: Preprocess the tactile signal and perform short-time Fourier transform (STFT) on the tactile sequence to obtain the spectrum of the tactile sequence. When performing STFT transformation, it is necessary to select a suitable window function and window size and satisfy the constraint of "Constant OverLap Add" (COLA) to ensure that there is no information loss during STFT transformation. In this way, when restoring touch, it can be restored to the tactile sequence through ISTFT transformation. The spectrum is calculated by the following formula to obtain the amplitude spectrum and phase spectrum Then splice them to get the tactile signal

[0082]

[0083] Finally, the RGB image and tactile signals Splice into five-dimensional data in the channel dimension Where n = H × W × 5 represents the size of the image tactile signal pair. During the processing, the one-to-one pairing of the image and the tactile signal is maintained;

[0084] Step 2: Input the preprocessed image and tactile signal into the cross-modal joint encoder. First, the input data is processed in blocks, which can reduce the length of the subsequent embedding sequence and more effectively capture long-distance dependencies. Then, it is input into the hierarchical feature extraction module and the signal-to-noise ratio adaptation module in turn, and finally the hierarchical fusion feature is obtained, such as Figure 2 As shown in the figure, the hierarchical feature extraction module extracts and fuses the multi-level features of the image-tactile signal, and the signal-to-noise ratio adaptive module dynamically adjusts the multi-level fusion features generated by the hierarchical feature extraction module according to the real-time channel conditions to adapt to the changing channel environment.

[0085] Step 2-1: Input the preprocessed image and tactile signal into the cross-modal joint encoder. Divide the input data into blocks, which can reduce the length of the subsequent embedding sequence and more effectively capture long-distance dependencies. In addition, cutting the image into small blocks can retain local structural information. Specifically, the data pair is divided into non-overlapping blocks (patches) of size 4×4. Through this division method, the feature dimension of each patch is 4×4×5=80 dimensions. At this time, the size of the data tensor is

[0086] Step 2-2: Then the divided data is input into multiple hierarchical feature extraction modules and signal-to-noise ratio adaptive modules. The hierarchical feature extraction module is represented by FE e (·). Among them, the signal-to-noise ratio adaptive module is denoted as SD e (·), connected after the first and last feature extraction components, respectively, to learn feature representations of shallow and deep semantics with different signal-to-noise ratios, such as Figure 2Specifically, assuming that the number of feature extraction components in the encoder is L, and the input of the jth feature extraction component is t j ,j∈{1,2,...,L}. The continuous calculation process is as follows:

[0087]

[0088] First, the tactile-image signal pair Divide it into non-overlapping small blocks in both length and width to get t1, and input it into the hierarchical feature extraction module to get the first layer feature representation Next, the first layer of feature representation is input into the signal-to-noise ratio adaptation module to obtain t2, and then t2 is input into multiple feature extraction components. After passing through the last layer of feature extraction components, the deep features are obtained, and then input into the last layer of signal-to-noise ratio adaptation layer to obtain the remapped features e2.

[0089] Step 2-3, wherein the hierarchical feature extraction module FE e (·) Includes merging layers and SwinTransformer modules to learn hierarchical features of tactile-image signals. The merging layer is mainly responsible for downsampling and upscaling to produce features at different levels, while the SwinTransformer block mainly performs hierarchical feature learning. The first merging layer connects the features of each group of 2×2 adjacent patches in the channel dimension. Through such processing, the feature resolution will be downsampled by 2 times, and the feature dimension will increase by 4 times due to the connection operation. Therefore, a fully connected layer is applied to the connected features to unify the output dimension to twice the original dimension, that is, 2C. At this time, the tensor size is Then, it passes through the SwinTransformer block and merging layer multiple times to obtain a hierarchical feature representation. The last merging layer of the cross-modal joint encoder is responsible for mapping the deep feature information to the same dimension as the channel channel size. Therefore, unlike the previous merging layers, after connecting the adjacent patch features, a linear layer is applied to transform the output dimension to the channel dimension b. At this time, the tensor size is

[0090] The Swin Transformer module is the key to extracting features. The Swin Transformer module will be introduced in detail below. Figure 3 As shown in Figure 1, two consecutive SwinTransformer modules, each module consists of a normalization (LayerNorm, LN) layer, a multi-head self-attention module, a residual connection, and a multi-layer perceptron (MLP). The two connected SwinTransformer modules use a window-based multi-head self-attention module (W-MSA) and a shifted window-based multi-head self-attention module (SW-MSA).

[0091] In the standard ViT (Vision Tansformer), global attention is used to calculate the relationship between the target token and all other tokens, which increases the computational complexity. The Swin Transformer module is built based on displacement windows. Each tensor input to the Swin Transformer will be evenly divided into several non-overlapping windows, and it only calculates self-attention within the local window. Because the number of patches within the window is much smaller than the number of patches in the image, the amount of calculation is greatly reduced. On the other hand, WMSA only calculates self-attention within the window, and there is a lack of information exchange between non-overlapping windows. In order to improve the model's ability to build global relationships, it is also necessary to build SW-MSA, which introduces displacement windows to achieve cross-window information exchange and increase the receptive field.

[0092] The shifted window method will cause a problem. Since the window size is not aligned with the tensor size, after the shift, the number of windows will increase, and the size of the elements in each window will be different. The paper "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" proposes a method of using cyclic shift plus masking to keep the number of windows unchanged and the elements uniform after the shift.

[0093] Specifically, the shift window method is implemented between two consecutive SwinTransformer modules. The first module performs WMSA operation to calculate the attention within the window, and the second module performs SW-MSA operation to move the window to calculate the attention. Assume that the input of the i-th Swin Transformer module is u i , the specific calculation process is as follows:

[0094]

[0095] Among them, LN stands for normalization layer, WMSA stands for window multi-head self-attention module, MLP stands for multi-layer perceptron, and SW-MSA stands for shifted window multi-head self-attention module. Represents the output of the i-th Swin Transformer module. The attention calculation process is as follows:

[0096]

[0097] The input token sequence will generate three matrices: query, key, and value, denoted as Q, K, and V. d represents the dimension of query or key. B represents the relative position deviation.

[0098] Step 2-4, the signal-to-noise ratio adaptation module SD(·) is the key to ensure that the model adapts to different channel levels, which includes global pooling, feedforward network, such as Figure 4 As shown in Figure 1, global pooling is used to capture the full-size information of the input features, aggregate local features into global features, connect the signal-to-noise ratio information with the pooled features, and input the feed-forward network (FFN) to generate scaling factors. These scaling factors are combined with the hierarchical feature extraction module FE e The output features of (·) are multiplied to obtain weighted features, which are then used in subsequent encoding and transmission processes. The purpose of multiplication is to adjust the importance of features according to different SNR conditions so that the model can handle channel noise and transmit data more effectively.

[0099] First, the output of the hierarchical feature extraction component is o={o1,o2,...,o c},o ipq Represents the element whose i-th channel is located at (p,q). After global average pooling P(·), each feature map is pooled in the channel dimension, so that global information can be extracted. The global pooling operation transforms each two-dimensional feature map o on the channel i Compressed into a single value s i , represents the average activation intensity of the entire feature map.

[0100]

[0101] Then, these global information are concatenated with the current SNR value to obtain It is input into a feed-forward network FFN(·), the output of which is a scaling factor that determines the weight of each element in the next feature map. The feed-forward network consists of multiple linear layers, except for the last layer whose activation function is Sigmoid, the remaining layers use Relu activation function. This ensures that the output scaling factor is between 0 and 1.

[0102]

[0103] Finally, Compared with the original input features before pooling Multiply to get the features adjusted according to the channel conditions Since the two dimensions are different, the data of each channel of factor is copied before multiplication, thus expanding the dimension to H×W×c.

[0104] The SNR adaptive module can perform weighted adjustment on each element in the original feature map by using the scaling factor predicted by the feedforward network. The SNR adaptive module does not change the size of the input features. Specifically, each element in the feature map is multiplied by the corresponding scaling factor to achieve recalibration of the features. If the scaling factor is close to 0, it means that the corresponding feature is not very important under the current SNR conditions and should be suppressed; if the scaling factor is close to 1, the feature remains unchanged or enhanced. Unlike traditional binary masks, the scaling factor is equivalent to a soft mask. The scaling factor is a value between 0 and 1, which is used to proportionally adjust the weight of the feature instead of simply switching it on and off. This method provides a more delicate and continuous control method, allowing the model to adjust the importance of features at a finer granularity based on the current channel conditions and image content.

[0105] Step 3: After the hierarchical fusion features are transmitted through the channel, they enter the cross-modal joint decoder. The cross-modal joint decoder adopts a symmetrical design similar to the encoder structure, including a hierarchical feature extraction module, a signal-to-noise ratio adaptation module, and a signal reconstruction module, such as Figure 2 As shown in the figure, the hierarchical feature extraction module decodes the tactile low-dimensional features from the hierarchical fusion features, and the signal-to-noise ratio adaptive module dynamically adjusts the low-dimensional features according to the channel changes. Finally, the signal reconstruction module converts the low-dimensional features into tactile spectrum, and the tactile spectrum is converted into tactile sequence through inverse short-time Fourier transform (ISTFT), thereby completing the tactile signal reconstruction.

[0106] Step 3-1: The hierarchical fusion features are transmitted through the channel and input into the cross-modal joint decoder. Its structure corresponds to the cross-modal joint encoder and consists of multiple hierarchical feature extraction modules. Signal-to-noise ratio adaptive module SD1 d (·) and signal reconstruction modules. At this point, the hierarchical feature extraction module consists of an expansion layer and a SwinTransformer module. Unlike the merging layer used in the encoder, the expansion layer is used in the decoder to upsample the extracted deep features. The expansion layer reshapes the feature maps of adjacent dimensions into feature maps of larger sizes (2× upsampling) and correspondingly reduces the feature dimension to half of the original dimension, completing the dimensionality reduction of the feature.

[0107] Step 3-2: The channel signal is input into multiple hierarchical feature extraction modules to obtain features at different levels. Similarly, the signal-to-noise ratio adaptive module is connected after the first and last hierarchical feature extraction modules.

[0108] Specifically, assuming that the number of hierarchical feature extraction modules in the decoder is L, the input to the jth hierarchical feature extraction module according to the data flow is t j,j∈{1,2,...,L}. The continuous calculation process is as follows:

[0109]

[0110] The signal transmitted from the channel Input the hierarchical feature extraction module to obtain the first layer of dimensionality reduction feature representation Next, the signal-to-noise ratio adaptive module is input to obtain t2. After that, t2 is reduced in dimension by multiple hierarchical feature extraction modules, and then passed through a signal-to-noise ratio adaptive layer again to perform feature scaling on the shallow features after the final dimension reduction to obtain the remapped feature e2. Secondly, it is scaled 4 times by an expansion layer to convert the feature size into

[0111] Step 3-3: Input the reduced-dimensional features into the signal reconstruction module, and map them from the feature domain to a time-frequency matrix containing the amplitude and phase information of the tactile sequence through multiple convolutional layers. In this stage, the multi-level features are input into multiple convolutions to keep the size unchanged and realize the scaling of the number of channels. Finally, the number of channels is scaled to 2, accurately mapping to the time-frequency matrix of the amplitude and phase information of the tactile sequence. Finally, the restored tactile sequence is obtained through the ISTFT transformation.

[0112] Step 4: jointly train the cross-modal joint encoder and the cross-modal joint decoder according to the training data to obtain optimal model parameters;

[0113] Step 4-1: In the training phase, the model receives tactile image signal pairs and channel signal-to-noise ratio information for training and learns the mapping relationship under different signal-to-noise ratio levels. The Charbonnier loss is selected as the loss function of the optimization process. The calculation formula is shown in the formula:

[0114]

[0115] Among them, h j represents the tactile signal recovered by the decoder, h j Represents the real tactile signal, ε=10 -3 is a constant. The existence of the constant ε ensures that the gradient will not be too small when it approaches zero, and the square root helps to avoid the gradient from vanishing. The Charbonnier loss function is more robust to outliers. Compared with the mean square error (L2 loss), it is insensitive to large residuals, which makes the model more stable during training, less susceptible to noise, easier to converge quickly, and the loss calculation is relatively simple.

[0116] Step 4-2: The cross-modal joint encoder and the cross-modal joint decoder are jointly trained as a whole network, the gradient is calculated by the Adam optimization algorithm, and the network parameters are updated by back propagation. The expression is as follows:

[0117]

[0118] Among them, m t and v t are unbiased estimates of the first and second order moments, respectively. t is the gradient of the parameter, beta1 and beta2 are the decay rates of the first-order moment estimate and the second-order moment estimate, respectively. μ is the learning rate, and ε is a small constant used to ensure that the denominator is not zero. θ t and θ t+1 They are the network parameters before and after the update, respectively. The network parameters here refer to the network parameters composed of the cross-modal joint encoder and the cross-modal joint decoder. The learning rate is selected as 1×10-4 and the number of iterations is 1000. After the iteration, the network parameters are determined and the model training is completed.

[0119] Step 5: Input the tactile and image signals to be tested into the optimal cross-modal joint encoder obtained in the above steps to obtain hierarchical fusion features. Then, after channel transmission, the received hierarchical fusion features are input into the cross-modal joint decoder to generate the target tactile spectrum, and the target tactile signal is obtained after ISTFT transformation.

[0120] Step 5-1, selecting the trained optimal cross-modal source encoder, channel encoder-decoder and tactile decoder network model;

[0121] Step 5-2: Input the tactile-image signal pair to be tested into the model. First, it passes through the cross-modal joint encoder to generate a hierarchical fusion feature. Then, after transmission through the channel, it is decoded by the cross-modal joint decoder and the spectrum of the target tactile signal is reconstructed. Finally, the desired tactile signal is obtained through the ISTFT transform.

[0122] Embodiment 2, an embodiment of the present invention, which is different from the above embodiment in that:

[0123] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0124] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0125] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0126] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0127] Example 3, reference Figure 5-Figure 7 , which is an embodiment of the present invention, provides a tactile signal reconstruction method for signal-to-noise ratio adaptive joint optimization of source channels. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through simulation experiments.

[0128] This embodiment uses the AUdataset dataset for experiments, which was collected by Lasse et al. from the Department of Electrical and Computer Engineering, Aarhus University. The dataset contains multimodal signals of 63 objects: visual, kinesthetic and tactile (audio vibration signals). The objects selected in the dataset are mainly toys and household items. Each object contains 3 pictures from different angles and 5 recorded tactile vibration signals. Among them, there are objects of similar shapes and colors (visual ambiguity) and objects of similar materials and textures but different colors (tactile ambiguity). These objects containing visual ambiguity and tactile ambiguity will reflect the different effects of vision and touch on perception. This example selects 62 available objects, expands the 3 images of each object to 5 (for example, randomly adjusts the brightness and contrast of the picture, flips the image), and forms 310 groups of samples with 5 recorded tactile vibration signals. In the dataset, 80% are selected for training, and the remaining 20% ​​are used for testing and performance evaluation.

[0129] First, the method of the present invention is compared with a simple tactile reconstruction method, that is, the method of the present invention needs to consider channel interference while the simple tactile reconstruction method only considers the signal reconstruction on the terminal side. The following three tactile reconstruction methods are selected for experimental comparison:

[0130] Existing method 1:

[0131] Literature: "Cross-ModalDataGenerationUsingResidue-FusionGANWithFeature-MatchingandPerceptualLosses"

[0132] (Authors Cai S, Zhu K, Ban Y, et al) used the cGAN (conditional-GAN) structure and residual-fusion (RF) module, and trained the model with additional feature matching (FM) and perceptual loss to achieve mutual generation between visual images and tactile spectra.

[0133] Existing method 2:

[0134] The paper "Toward Image-to-Tactile Cross-Modal Perception for Visually Impaired People" (author: Liu H, Guo D, Zhang X, et al) relies on DiscoGAN to learn the conversion between images and spectrograms. By using the trained DX and DY as discriminators to double constrain the generated results, and adding an auxiliary classification layer to the original discriminator, where category information is needed, the model can learn the mutual conversion between the visual domain and the tactile domain.

[0135] Existing method 3: Literature

[0136] “Learning cross-modal visual-tactile representation using ensembled generative advancerarial networks” (author: Li X, Liu H, Zhou J, et al) is based on deep learning and uses the texture attributes of the image as input. After obtaining the image category information, it is used as the selection condition of GAN, thereby realizing cross-modal conversion from texture images to tactile signals through corresponding GAN training.

[0137] The present invention: the method of this embodiment.

[0138] The experiment uses similarity (SIM) as an evaluation indicator to evaluate the effect of cross-modal generation. SIM is used to measure the similarity between the real tactile signal and the reconstructed tactile signal, and is defined as:

[0139]

[0140] Among them, the exponent in the denominator is (h i ,h i ′) is h i and h i ′The Chebyshev distance between the waveforms. That is, the maximum distance between two waveform points. The range of SIM is [0,1]. When the difference between the two is small, SIM is closer to 1 and the two waveforms are more similar, and vice versa.

[0141] Table 1 is the experimental results of the present invention.

[0142]

[0143] From Table 1 and Figure 5 It can be seen that compared with the above-mentioned state-of-the-art methods, our proposed method has obvious advantages. All other comparative model inputs are only visual signals, while our model fuses vision and touch, extracts high-level semantic information of the two modalities of the same object, and better extracts the common information between modalities and the unique information within the modality, thereby better restoring the tactile signal. Figure 5 The tactile reconstruction results compared with other comparison methods are shown. Figure 6 The tactile spectrum and sequence reconstruction result diagram of the method of the present invention are shown.

[0144] Secondly, in order to further verify the adaptability of the method of the present invention to the signal-to-noise ratio, the tactile reconstruction effects of the method of the present invention and the comparative method are compared under different signal-to-noise ratios. Here, the comparative method uses a self-encoder-decoder structure composed of multiple linear layers and convolutional layers to replace the cross-modal joint codec structure in the present invention. Then, experiments are conducted under different test signal-to-noise ratios. Figure 7 Under different signal-to-noise ratios, the method of the present invention can reconstruct tactile signals with high quality, showing the adaptive characteristics of the signal-to-noise ratio change, while the tactile reconstruction effect of the comparison method under low signal-to-noise ratio is not ideal.

[0145] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A tactile signal reconstruction method for signal-to-noise ratio adaptive joint optimization of source channels, characterized in that: include: Preprocessing the image signal corresponding to each object and the tactile vibration signal generated by sliding based on the tactile-image database; The preprocessed image and tactile signal are input into the cross-modal joint encoder, the input data is processed in blocks, and then input into the hierarchical feature extraction module and the signal-to-noise ratio adaptive module in sequence to obtain the hierarchical fusion features; After being transmitted through the channel, the hierarchical fusion features enter the cross-modal joint decoder, which includes a hierarchical feature extraction module, a signal-to-noise ratio adaptation module, and a signal reconstruction module; The hierarchical feature extraction module decodes the low-dimensional tactile features from the hierarchical fusion features. The signal-to-noise ratio adaptive module dynamically adjusts the low-dimensional features according to the channel changes. The signal reconstruction module converts the low-dimensional features into a tactile spectrum. The tactile spectrum is converted into a tactile sequence through an inverse short-time Fourier transform (ISTFT), thereby completing the tactile signal reconstruction. The cross-modal joint encoder and the cross-modal joint decoder are jointly trained according to the training data to obtain the optimal model parameters; The tactile and image signals to be tested are input into the optimal cross-modal joint encoder to obtain hierarchical fusion features. After channel transmission, the received hierarchical fusion features are input into the cross-modal joint decoder to generate the target tactile spectrum, and the target tactile signal is obtained through ISTFT transformation.

2. The method for tactile signal reconstruction by adaptively joint optimizing the signal-to-noise ratio of the source channel according to claim 1, characterized in that: The preprocessing includes cutting the image into H×W size, which is expressed as: Among them, v j Represents the jth RGB three-channel image of size H×W in the dataset; The tactile signal is preprocessed, and the tactile sequence is subjected to short-time Fourier transform (STFT) to obtain the spectrum of the tactile sequence, which is expressed as: Pass the spectrum through and , and get the amplitude spectrum and phase spectrum The amplitude spectrum and phase spectrum are spliced ​​to obtain the tactile signal Convert RGB image and tactile signals Splice into five-dimensional data in the channel dimension Wherein, n=H×W×5 represents the size of the image tactile signal pair, and during the processing, the one-to-one pairing of the image and the tactile signal is maintained.

3. The method for tactile signal reconstruction by adaptively joint optimizing the signal-to-noise ratio of the source channel according to claim 2, characterized in that: The block processing includes dividing the data pairs into non-overlapping blocks of size 4×4; The feature dimension of each patch is 4×4×5=80, and the size of the data tensor is The divided data is input into multiple hierarchical feature extraction modules and signal-to-noise ratio adaptive modules; The hierarchical feature extraction module is FE e (·), the signal-to-noise ratio adaptive module is SD e (·), connected after the first and last feature extraction components, respectively; Assume that the number of feature extraction components in the encoder is L, and the input of the jth feature extraction component is t j ,j∈{1,2,...,L}, the continuous calculation process is expressed as, The tactile-image signal pair Divide it into non-overlapping small blocks in both length and width to get t1, and input it into the hierarchical feature extraction module to get the first layer feature representation The first layer of feature representation is input into the signal-to-noise ratio adaptive module to obtain t2, t2 is input into multiple feature extraction components, deep features are obtained after passing through the last layer of feature extraction components, and input into the last layer of signal-to-noise ratio adaptive layer to obtain the remapped feature e2.

4. The method for tactile signal reconstruction by adaptively joint optimizing the signal-to-noise ratio of the source channel according to claim 3, characterized in that: The hierarchical feature extraction module FE e (·) Includes merging layers and Swin Transformer modules to learn hierarchical features of tactile-image signals; The merging layer is responsible for downsampling and dimension upscaling to produce features at different levels, while the SwinTransformer block performs hierarchical feature learning; The first merging layer connects the features of each group of 2×2 adjacent patches in the channel dimension, downsampling the feature resolution by 2 times and increasing the feature dimension by 4 times; Applying a fully connected layer on the concatenated features unifies the output dimension to twice the original dimension, i.e. 2C, and the tensor size is After multiple passes through the Swin Transformer blocks and merging layers, a hierarchical feature representation is obtained; The last merging layer of the cross-modal joint encoder is responsible for mapping the deep feature information to the same dimension as the channel channel size. After connecting the adjacent patch features, a linear layer is applied to transform the output dimension to the channel dimension b. At this time, the tensor size is 5. The method for tactile signal reconstruction by adaptively joint optimizing the signal-to-noise ratio of the source channel according to claim 4, characterized in that: The signal-to-noise ratio adaptation module SD(·) includes a global pooling and a feed-forward network; The global pooling is used to capture the full-size information of the input features, aggregate the local features into global features, connect the signal-to-noise ratio information with the pooled features, and input the feedforward network to generate a scaling factor. The scaling factor is combined with the hierarchical feature extraction module FE e (·) to obtain the weighted features; The output of the feature extraction component is o={o1,o2,...,o c },o ipq Indicates the element whose i-th channel is located at (p,q); After global average pooling P(·), each feature map is pooled in the channel dimension to extract global information The global pooling operation transforms each two-dimensional feature map o on the channel i Compressed into a single value s i , represents the average activation intensity of the entire feature map, expressed as, Concatenate the global information with the current SNR value to get Input it into the feedforward network, expressed as, Will Compared with the original input features before pooling Multiply to get the features adjusted according to the channel conditions Since the two dimensions are different, the data of each channel of factor is copied before multiplication, thus expanding the dimension to H×W×c.

6. The method for tactile signal reconstruction by adaptively joint optimizing the signal-to-noise ratio of the source channel according to claim 5, characterized in that: The cross-modal joint decoder uses an expansion layer to upsample the extracted deep features. The expansion layer reshapes the feature maps of adjacent dimensions and reduces the feature dimension to half of the original dimension, thereby completing the dimensionality reduction of the features. The channel signal is input into multiple hierarchical feature extraction modules to obtain features at different levels. The signal-to-noise ratio adaptive module is connected after the first and last hierarchical feature extraction modules; Assume that the number of hierarchical feature extraction modules in the cross-modal joint decoder is L, and the input to the jth hierarchical feature extraction module according to the data flow is t j ,j∈{1,2,...,L}, the continuous calculation process is expressed as, The signal transmitted from the channel Input the hierarchical feature extraction module to obtain the first layer of dimensionality reduction feature representation The input signal-to-noise ratio adaptive module obtains t2. After t2 is reduced in dimension by multiple hierarchical feature extraction modules, it passes through a signal-to-noise ratio adaptive layer again to perform feature scaling on the shallow features after the final dimension reduction to obtain the remapped feature e2. It is scaled 4 times by an expansion layer to convert the feature size into The feature input signal reconstruction module after dimensionality reduction is mapped from the feature domain to the time-frequency matrix containing the amplitude and phase information of the tactile sequence through multiple convolutional layers, which is expressed as, The multi-level features are input into multiple convolutions to keep the size unchanged and achieve the scaling of the number of channels. The number of channels is scaled to 2 and mapped to the time-frequency matrix of the amplitude and phase information of the tactile sequence. The restored tactile sequence is obtained through ISTFT transformation.

7. The method for tactile signal reconstruction by adaptively joint optimizing the signal-to-noise ratio of the source channel according to claim 6, characterized in that: The joint training includes selecting Charbonnier loss as the loss function of the optimization process, expressed as, Among them, h j represents the tactile signal recovered by the decoder, h j Represents the real tactile signal, ε=10 -3 is a constant; The cross-modal joint encoder and the cross-modal joint decoder are jointly trained as a whole network, the gradient is calculated by the Adam optimization algorithm, and the network parameters are updated by back propagation. The expression is as follows: Among them, m t and v t are unbiased estimates of the first and second order moments, respectively. t is the gradient of the parameter, beta1 and beta2 are the decay rates of the first-order moment estimate and the second-order moment estimate, respectively. μ is the learning rate, and ε is a small constant used to ensure that the denominator is not zero. θ t and θ t+1 They are the network parameters before and after the update, respectively. The network parameters here refer to the network parameters composed of the cross-modal joint encoder and the cross-modal joint decoder. The learning rate is selected as 1×10-4 and the number of iterations is 1000. After the iteration, the network parameters are determined and the model training is completed.

8. The method for tactile signal reconstruction by adaptively joint optimizing the signal-to-noise ratio of the source channel according to claim 7, characterized in that: The target tactile signal reconstruction includes generating a hierarchical fusion feature through a cross-modal joint encoder, decoding by a cross-modal joint decoder after channel transmission, and reconstructing the frequency spectrum of the target tactile signal, and obtaining the desired tactile signal through ISTFT transformation.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method for tactile signal reconstruction by adaptively jointly optimizing the signal-to-noise ratio of a source channel according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for tactile signal reconstruction by adaptively jointly optimizing a source channel through a signal-to-noise ratio according to any one of claims 1 to 8 are implemented.

Images