A tactile signal encoding method and system based on cross-sensory masking of visual haptics

Abstract

The application discloses a kind of based on visual tactile cross-sensory masking tactile signal encoding method and system, it is related to cross-sensory tactile signal perception computing coding technical field, including: obtaining original three-axis tactile signal and tactile signal corresponding video frame, original three-axis tactile signal is framed, three-axis dimension reduction and normalization processing is carried out, and normalized tactile frame is obtained, tactile signal corresponding video frame is extracted, and visual motion speed is obtained;Visual motion speed is input into the visual tactile cross-sensory perception threshold model established in advance, and the frequency domain perception weight obtained is output, and normalized tactile frame and frequency domain perception weight are input into the perception driving frequency enhancement and residual vector quantization model, and the difference between normalized reconstruction frame and normalized tactile frame and RVQ code word index are output;Residual signal is calculated based on the difference between normalized reconstruction frame and normalized tactile frame, and perception weighting, dead zone processing and adaptive quantization are carried out, to generate residual quantization index, and RVQ code word index and residual quantization index are entropy coded based on preset auxiliary information, to obtain output bit stream, while guaranteeing tactile subjective perception quality, the efficient compression of tactile signal is realized.

Description

Technical Field

[0001] This invention relates to the field of cross-sensory tactile signal perception computational coding technology, specifically a tactile signal coding method and system based on visual-tactile cross-sensory masking. Background Technology

[0002] The bit rate of tactile signals cannot be ignored in multi-channel scenarios. Taking a single vibration tactile channel as an example, if the sampling rate is 2800Hz and the quantization depth is 16bit, the raw bit rate is approximately 44.8kbps. If a user simultaneously wears a vest, gloves, or fingertip actuator containing dozens of tactile channels, and the system supports multi-user collaboration, the raw tactile bit rate may reach a high level. Since visual signals typically have higher spatial and temporal resolution, occupy the majority of bandwidth in communication systems, and play a dominant role in the overall quality of human perception, excessively high tactile bit rates will crowd out the available resources of the visual stream, thus affecting the overall experience of multi-sensory communication.

[0003] Existing haptic technologies mainly focus on haptic effect generation, haptic effect storage, conversion of audio or multimedia content into haptic effects, and synchronized playback of haptic data with audio and video media streams. Therefore, existing technologies do not address low-bitrate compression encoding of continuous vibration haptic signals, resulting in the inability to guarantee the quality of subjective haptic perception while achieving efficient compression of haptic signals. Summary of the Invention

[0004] To address the shortcomings mentioned in the background art, the present invention aims to provide a tactile signal encoding method and system based on visual-tactile cross-sensory masking.

[0005] Firstly, the objective of this invention can be achieved through the following technical solution: a tactile signal encoding method based on visual-tactile cross-sensory masking, the method comprising the following steps: The original triaxial tactile signal and the corresponding video frame are obtained. The original triaxial tactile signal is segmented, reduced in dimensionality, and normalized to obtain the normalized tactile frame. The visual motion speed is obtained by extracting the corresponding video frame. The visual motion speed is input into a pre-established visual-tactile cross-sensory perception threshold model, and the frequency domain perception weight is output. The normalized tactile frame and the frequency domain perception weight are input into the perception-driven frequency enhancement and residual vector quantization model, and the difference between the normalized reconstructed frame and the normalized tactile frame and the RVQ codeword index are output. The residual signal is calculated based on the difference between the normalized reconstructed frame and the normalized tactile frame. The residual signal is then subjected to perceptual weighting, dead zone processing, and adaptive quantization to generate a residual quantization index. The RVQ codeword index and the residual quantization index are then entropy encoded based on preset auxiliary information to obtain the output bit stream, which serves as the encoding result of the tactile signal.

[0006] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the three-axis dimensionality reduction processing procedure, including: The DFT321 dimensionality reduction algorithm is introduced to map the high-dimensional spatial signal in the original triaxial tactile signal into a one-dimensional signal that is easier to process. Specifically, according to the DFT321 dimensionality reduction algorithm, the spectrum... The expression for phase θ is as follows: Where f represents frequency, A1 represents the discrete Fourier transform of the original tactile frame along the X-axis, A2 represents the discrete Fourier transform of the original tactile frame along the Y-axis, and A3 represents the discrete Fourier transform of the original tactile frame along the Z-axis. The one-dimensional frame x can be derived from the preset spectrum and phase.

[0007] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the process of inputting the visual motion velocity into a pre-established visual-tactile intersensory perception threshold model and outputting frequency domain perception weights, comprising: Threshold curve for a given visual motion speed Find the reciprocal to obtain the tactile sensitivity function. Then, the sensitivity function is normalized using its maximum value within the frequency range to obtain a perceptual weight with a value range of 0 to 1. The formula is as follows: Where ϵ is a regularization constant introduced to avoid division by zero. express The maximum value across the entire frequency range; After scale extraction, the residual signal e is reshaped in the frequency domain, thus focusing coding resources on the sensing-related components. The time-domain residual is transformed using a Fast Fourier Transform, and a smoothing factor μ is introduced to construct a softened weight vector w to mitigate the excessive attenuation of high-frequency residual components when directly using the original sensing weight vector u in the main path. The softened weights are constructed based on the obtained sensing weight vector w. As the frequency domain sensing weight, the formula is as follows: .

[0008] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: a processing procedure for enhancing the perception driving frequency and the residual vector quantization model, comprising the following steps: For the obtained normalized haptic frame x, a data embedding layer consisting of one-dimensional convolutions will input a single channel. Projected into a higher-dimensional latent space, producing Where D represents the number of feature channels, and a Fast Fourier Transform is applied independently to each D channel to obtain... ; In the frequency domain, a learnable complex-valued parameter kernel is introduced and optimized during end-to-end training. By performing element-wise multiplication between Q and R, a learnable complex-valued parameter kernel R is introduced as an adaptive frequency domain equalizer to perform element-wise weighting on the spectral representation Q, resulting in a frequency-enhanced spectral representation: The kernel function R serves as an adaptive frequency domain equalizer. The normalized perceptual weight vector w is broadcast along the feature dimension to form a perceptual mask W, and then... Further perform perception weighting: By performing an inverse fast Fourier transform on the frequency domain feature Y, the time domain is restored and a frequency-enhanced frame y is obtained. Then, y is input into a depth encoder consisting of four cascaded one-dimensional convolutional layers to progressively compress the time dimension and generate a compact latent tactile representation z.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: a residual vector quantization process for the perception driving frequency enhancement and residual vector quantization model, comprising the following steps: Residual vector quantization decomposes the original quantization task into L consecutive residual quantization stages, given the input features z and a set of independent codebooks. Each codebook contains K learnable codewords. The input to the l-th quantizer is the residual, where 1 ≤ l ≤ L, and its initial value is... ; The most recent codeword is selected from the current codebook, where represents the Euclidean norm. The residual is then updated and passed to the next stage. The final reconstructed feature ẑ is obtained by summing the codebook vectors selected in all L stages. The number of residual stages is set to L=3, and each codebook contains K=128 codewords. Encoding a single feature vector requires only three index values, totaling [number missing]. In the first training epoch, all codebooks are initialized using the K-means algorithm: Where K represents the number of discriminator models, and x' represents the normalized frame. Indicates normalized reconstructed frame This represents the output of the k-th discriminator in the last layer.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the sensing drive frequency enhancement and residual vector quantization model is performed end-to-end. Training, the loss function used for training is as follows: in, The MSE loss is used to constrain the energy difference in the time domain between the normalized input frame and the normalized reconstructed frame. SSIM loss is used to maintain the consistency of the local temporal envelope and amplitude structure. The perception loss is used to increase the reconstruction weights of the perception-sensitive frequency bands based on the perception mask W. To combat the loss, a multi-scale discriminator is used to make the reconstructed tactile waveform more realistic. The feature matching loss is used to constrain the consistency of the intermediate feature maps of the discriminator. The loss commitment is used to constrain the encoder output to be close to the selected codeword.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the MSE loss. This item normalizes the input signal. With reconstructed signal To minimize the energy difference in the time domain, the MSE loss is defined as: Where N is the total number of samples in the signal; SSIM loss The loss term preserves the structural consistency of the local temporal envelope and amplitude contrast: Perceived loss By incorporating a predefined perceptual mask W, this loss term prompts the model to prioritize spectral reconstruction of perceptually sensitive frequency bands, while imposing a lower penalty on errors in the visual mask region: in This represents the complex coefficients of the k-th frequency band. It is a regularization constant introduced to prevent gradient vanishing in the mask region; Combating losses A multi-scale discriminator operating at a preset scale is employed to facilitate the generation of realistic tactile waveforms: in This represents the output of the discriminator at the j-th scale; Feature matching loss minimize The distance between the intermediate feature maps of the discriminator ensures that the generated signal remains close to the real signal in the representation space: in This represents the sum of the absolute differences of all elements in the feature map; Loss commitment Limiting the encoder output z to remain close to the selected codeword improves numerical stability during quantization. in This indicates that the gradient operator is stopped.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the calculation of the residual signal based on the difference between the normalized reconstructed frame and the normalized haptic frame is as follows: Compare the original normalized haptic frame x with the base normalized reconstructed frame. Subtracting them yields the residual signal: Where 'e' represents the residual details that are not fully represented by the underlying coding path; Extract the logarithmic scale index of the residuals and calculate the absolute peak value of the residuals in the current frame. After mapping it to the logarithmic field, it is uniformly quantized into an 8-bit scale index. : in This indicates a round-down operation. and It is a predefined safety boundary with dynamic range; The residual signal e is transformed to the frequency domain, and a smoothing factor μ is introduced to construct a softened weight vector w. The softened weights are then constructed based on the obtained perceptual weight vector w. , The softened weights are applied to the residual spectrum, and the residual after perceptual shaping is obtained through inverse transform. : in and This represents the Fast Fourier Transform and the Inverse Fast Fourier Transform. This represents element-wise multiplication, because The magnitude of the residual after shaping will be limited by the reconstruction scale s; An adaptive quantizer with dead zone is used to discretize the perceptually shaped residual. First, the dead zone threshold τ is determined based on the relative threshold and the lower limit of absolute noise. The dead zone threshold τ is determined by combining the relative threshold and the background absolute noise. in Indicates the dead zone ratio. This represents the absolute noise floor; samples with amplitudes below τ will be set to zero. Amplitude quantization is performed on non-zero residual samples. The residual quantization bit width B can be 0, 3, 4, 5, or 6, where B=0 indicates that the residual branch is omitted and only the base code rate level is used. When residual quantization is enabled, B contains 1 symbol bit, and the number of amplitude quantization levels M is: Normalize the non-zero residual samples according to the inverse quantization scale s to obtain α= / s, and α∈[0,1], then the amplitude quantization index I is generated. mag : I mag =round(α·M) Where round(·) represents rounding, and the amplitude quantization index I is... mag Together with symbol information, they serve as residual indexes. The residual vector quantization index and residual quantization index are then losslessly entropy encoded and encapsulated into a layered bitstream suitable for network transmission.

[0013] Secondly, in order to achieve the above objectives, the present invention discloses a tactile signal encoding system based on visual-tactile cross-sensory masking, comprising: The signal acquisition module is used to acquire the original triaxial tactile signal and the corresponding video frame of the tactile signal, perform frame segmentation, triaxial dimensionality reduction and normalization processing on the original triaxial tactile signal to obtain the normalized tactile frame, and extract the corresponding video frame of the tactile signal to obtain the visual motion speed. The signal processing module is used to input the visual motion speed into a pre-established visual-tactile cross-sensory perception threshold model and output the frequency domain perception weight. It inputs the normalized tactile frame and the frequency domain perception weight into the perception-driven frequency enhancement and residual vector quantization model and outputs the difference between the normalized reconstructed frame and the normalized tactile frame and the RVQ codeword index. The encoding output module is used to calculate the residual signal based on the difference between the normalized reconstructed frame and the normalized tactile frame, perform perceptual weighting, dead zone processing and adaptive quantization on the residual signal, generate a residual quantization index, and entropy encode the RVQ codeword index and the residual quantization index based on preset auxiliary information to obtain the output bit stream, which is used as the encoding result of the tactile signal.

[0014] The beneficial effects of this invention are: This invention establishes a visual-tactile cross-sensory perception threshold model, dynamically characterizing the impact of visual stimuli on tactile perception sensitivity using visual motion intensity in videos, thus providing a compression basis for tactile coding that better aligns with human perception characteristics. Through a perception-driven frequency enhancement module, cross-sensory perception weights are introduced into the tactile frequency domain coding process, suppressing perceptually insensitive spectral components under visual masking conditions and prioritizing the preservation of perceptually significant tactile details. Furthermore, residual vector quantization and perceptual scale residual quantization mechanisms further compensate for basic reconstruction errors. This invention effectively solves the problem of existing tactile coding methods relying solely on a single tactile modality and struggling to further compress tactile signals using visual-tactile perception correlation. In multi-sensory communication scenarios, it can provide low bitrate, low latency, and lossless or near-lossless tactile feedback, enhancing the realism and immersion of users in virtual reality, remote operation, and immersive multimedia services, achieving efficient compression of tactile signals while ensuring the quality of subjective tactile perception. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is the complete network structure diagram of the present invention; Figure 3 This is a schematic diagram of the model architecture of the present invention; Figure 4 This is a graph showing the SNR comparison results between the present invention and other methods; Figure 5 This is a comparison chart of the PSNR results of this invention with other methods; Figure 6 This is a comparison diagram of the ST-SIM results between the present invention and other methods; Figure 7 This is a flowchart of the system workflow of the present invention; Figure 8 This is a schematic diagram of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Example 1: like Figure 1 As shown, a tactile signal encoding method based on visual-tactile cross-sensory masking includes the following steps: S101: Obtain the original triaxial tactile signal and the corresponding video frame of the tactile signal, perform frame segmentation, triaxial dimensionality reduction and normalization on the original triaxial tactile signal to obtain the normalized tactile frame, and extract the corresponding video frame of the tactile signal to obtain the visual motion speed. The original triaxial vibration tactile signal undergoes preprocessing, including framing, triaxial dimensionality reduction, and normalization, to obtain normalized tactile frames. Simultaneously, visual motion intensity is extracted from video frames synchronized with the tactile signal, and corresponding frequency domain perception weights are generated based on a pre-constructed visual-tactile cross-sensory perception threshold model. Subsequently, the normalized tactile frames and frequency domain perception weights are input into a pre-constructed perception-driven frequency enhancement-residual vector quantization model. This deep neural network enhances perceptually sensitive tactile components in the frequency domain and suppresses perceptually insensitive tactile components under visual masking. Finally, the residual sequence of the normalized reconstructed frame and the original normalized frame, along with the corresponding RVQ codeword index, is output.

[0018] To achieve scalable control of transmission and processing latency, the continuously acquired triaxial vibration tactile signals are first written into a data buffer, and then a sliding window mechanism is used to divide them into discrete tactile frame sequences.

[0019] In this invention, the length of the sliding window is fixed at N=64 sampling points (approximately a 22.9ms time window at the current sampling rate). The multidimensional tactile feature frames processed by the sliding window will serve as the input for the next stage of spatial dimensionality reduction algorithm.

[0020] The process of three-axis dimensionality reduction includes: The DFT321 dimensionality reduction algorithm is introduced to map the high-dimensional spatial signal in the original triaxial tactile signal into a one-dimensional signal that is easier to process. Specifically, according to the DFT321 dimensionality reduction algorithm, the spectrum... The expression for phase θ is as follows: Where f represents frequency, A1 represents the discrete Fourier transform of the original tactile frame along the X-axis, A2 represents the discrete Fourier transform of the original tactile frame along the Y-axis, and A3 represents the discrete Fourier transform of the original tactile frame along the Z-axis. The one-dimensional frame x can be derived from the preset spectrum and phase.

[0021] S102: Input the visual motion speed into the pre-established visual-tactile cross-sensory perception threshold model, and output the frequency domain perception weight. Input the normalized tactile frame and the frequency domain perception weight into the perception-driven frequency enhancement and residual vector quantization model, and output the difference between the normalized reconstructed frame and the normalized tactile frame and the RVQ codeword index. The process of inputting visual motion velocity into a pre-established visual-tactile cross-sensory perception threshold model and outputting frequency domain perception weights includes: Threshold curve for a given visual motion speed Find the reciprocal to obtain the tactile sensitivity function. Then, the sensitivity function is normalized using its maximum value within the frequency range to obtain a perceptual weight with a value range of 0 to 1. The formula is as follows: Where ϵ is a regularization constant introduced to avoid division by zero. express The maximum value across the entire frequency range; After scale extraction, the residual signal e is reshaped in the frequency domain, thus focusing coding resources on the sensing-related components. The time-domain residual is transformed using a Fast Fourier Transform, and a smoothing factor μ is introduced to construct a softened weight vector w to mitigate the excessive attenuation of high-frequency residual components when directly using the original sensing weight vector u in the main path. The softened weights are constructed based on the obtained sensing weight vector w. As the frequency domain sensing weight, the formula is as follows: .

[0022] The construction and training process of the perception-driven frequency enhancement and residual vector quantization model includes: The perception-driven frequency enhancement and residual vector quantization model consists of a main path of feature frequency domain transformation and a perception parameter learning branch. First, a normalized frame x' in the time domain is input to the feature embedding layer for high-dimensional spatial mapping, and then converted into a frequency domain feature Q via a Fast Fourier Transform. This frequency domain feature can be clearly divided into perception-sensitive and perception-insensitive frequency bands, indicated by dashed lines, on the frequency axis. Simultaneously, in the bottom parameter calculation branch, the original continuous perception curve is transformed into discrete perception weights W through interpolation, used to quantitatively characterize the physiological perception sensitivity of different frequency bands. At the same time, the parameterized kernel R, initially flat, is dynamically trained through a kernel learning mechanism, evolving into a learnable parameterized kernel with specific frequency selection characteristics. Next, the frequency domain feature Q, perception weights W, and learnable parameterized kernel are fused through element-wise weighted multiplication to generate the modulated frequency domain feature Y. In feature Y, noise or redundant energy in perception-insensitive frequency bands is significantly suppressed, while features in perception-sensitive frequency bands are preserved and enhanced. Finally, the enhanced frequency domain features Y are reconstructed back into the time domain via inverse fast Fourier transform, outputting the final enhanced frame y. Simultaneously, end-to-end training is performed using a joint objective function consisting of six terms, improving model generalization and achieving a good balance between time-domain reconstruction fidelity and frequency-domain perception quality.

[0023] The tactile perception mechanism employed in this invention is primarily based on the frequency selectivity of human touch and the cross-sensory masking effect of visual-tactile perception. The human body's sensitivity to vibrations of different frequencies is not uniform, with the highest sensitivity to vibrations in the 150–200 Hz range, while sensitivity to vibrations below 30 Hz or above 500 Hz is relatively low. Simultaneously, when users view dynamic images with high motion intensity, visual information occupies more attentional resources, thus weakening their ability to perceive subtle tactile stimuli, causing the tactile perception threshold to increase with increasing visual motion intensity. Based on these characteristics, this invention uses optical flow amplitude to characterize video motion intensity and combines vibration frequency with the tactile perception threshold to construct a perception weight. This prioritizes tactile information within the human body's sensitive frequency bands while suppressing tactile components that are difficult to detect under strong visual stimulation, thereby reducing the signal transmission bit rate while ensuring subjective tactile quality.

[0024] The process of enhancing the perception-driven frequency and residual vector quantization model includes the following steps: For the obtained normalized haptic frame x, a data embedding layer consisting of one-dimensional convolutions will input a single channel. Projected into a higher-dimensional latent space, producing Where D represents the number of feature channels, and a Fast Fourier Transform is applied independently to each D channel to obtain... ; In the frequency domain, a learnable complex-valued parameter kernel is introduced and optimized during end-to-end training. By performing element-wise multiplication between Q and R, a learnable complex-valued parameter kernel R is introduced as an adaptive frequency domain equalizer to perform element-wise weighting on the spectral representation Q, resulting in a frequency-enhanced spectral representation: The kernel function R serves as an adaptive frequency domain equalizer. The normalized perceptual weight vector w is broadcast along the feature dimension to form a perceptual mask W, and then... Further perform perception weighting: By performing an inverse fast Fourier transform on the frequency domain feature Y, the time domain is restored and a frequency-enhanced frame y is obtained. Then, y is input into a depth encoder consisting of four cascaded one-dimensional convolutional layers to progressively compress the time dimension and generate a compact latent tactile representation z.

[0025] The process of residual vector quantization in the sensing-driven frequency enhancement and residual vector quantization model includes the following steps: Residual vector quantization decomposes the original quantization task into L consecutive residual quantization stages, given the input features z and a set of independent codebooks. Each codebook contains K learnable codewords. The input to the l-th quantizer is the residual, where 1 ≤ l ≤ L, and its initial value is... ; The most recent codeword is selected from the current codebook, where represents the Euclidean norm. The residual is then updated and passed to the next stage. The final reconstructed feature ẑ is obtained by summing the codebook vectors selected in all L stages. The number of residual stages is set to L=3, and each codebook contains K=128 codewords. Encoding a single feature vector requires only three index values, totaling [number missing]. In the first training epoch, all codebooks are initialized using the K-means algorithm: Where K represents the number of discriminator models, and x' represents the normalized frame. Indicates a normalized reconstructed frame. This represents the output of the k-th discriminator in the last layer.

[0026] The perception-driven frequency enhancement and residual vector quantization model is trained end-to-end, and the loss function used for training is as follows: in, The MSE loss is used to constrain the energy difference in the time domain between the normalized input frame and the normalized reconstructed frame. SSIM loss is used to maintain the consistency of the local temporal envelope and amplitude structure. The perception loss is used to increase the reconstruction weights of the perception-sensitive frequency bands based on the perception mask W. To combat the loss, a multi-scale discriminator is used to make the reconstructed tactile waveform more realistic. The feature matching loss is used to constrain the consistency of the intermediate feature maps of the discriminator. The loss commitment is used to constrain the encoder output to be close to the selected codeword.

[0027] MSE loss This item normalizes the input signal. With reconstructed signal To minimize the energy difference in the time domain, the MSE loss is defined as: Where N is the total number of samples in the signal; SSIM loss The loss term preserves the structural consistency of the local temporal envelope and amplitude contrast: Perceived loss By incorporating a predefined perceptual mask W, this loss term prompts the model to prioritize spectral reconstruction of perceptually sensitive frequency bands, while imposing a lower penalty on errors in the visual mask region: in This represents the complex coefficients of the k-th frequency band. It is a regularization constant introduced to prevent gradient vanishing in the mask region; Combating losses A multi-scale discriminator operating at a preset scale is employed to facilitate the generation of realistic tactile waveforms: in This represents the output of the discriminator at the j-th scale; Feature matching loss minimize The distance between the intermediate feature maps of the discriminator ensures that the generated signal remains close to the real signal in the representation space: in This represents the sum of the absolute differences of all elements in the feature map; Loss commitment Limiting the encoder output z to remain close to the selected codeword improves numerical stability during quantization. in This indicates that the gradient operator is stopped.

[0028] The perception-driven cross-sensory tactile coding model can be based on a one-dimensional convolutional encoder-decoder, a GRU recurrent network, or a Transformer network. Ultimately, a one-dimensional convolutional network is chosen. This model is suitable for extracting local texture, friction, and impact features from tactile vibrations. It can process 64-sample-point short frames in parallel, with lower computational cost and latency than GRU and Transformer. It is also easy to combine with frequency domain processing and vector quantization.

[0029] The model first divides the 2800 Hz, 16-bit triaxial haptic signal into 64 frames, which are then fused into a one-dimensional signal using a smoothing window and DFT321, followed by mean-standard deviation normalization. Subsequently, one-dimensional convolutional embedding, FFT, a trainable complex frequency domain kernel, and frequency weights generated based on video optical flow and visual-haptic perception thresholds are used to suppress frequency components that are difficult to perceive under visual motion masking. The enhanced features are encoded using four layers of one-dimensional convolutions and employ three levels of RVQ quantization. Each level of the codebook contains 128 codewords, requiring only 21 bits for a latent vector index. The decoding end reconstructs the signal using symmetric transpose convolution, and further details are rendered using 3-6 bit residual quantization and Huffman coding. Training is performed using the TUM dataset with five-fold cross-validation, with 80% training and 20% testing. A batch size of 256 and an initial learning rate of [missing value] are preferred. Gradient clipping 0.5, maximum training 500 epochs.

[0030] The maximum number of training epochs was set to 500 epochs primarily because the model simultaneously includes a one-dimensional convolutional encoder, a perceptual frequency enhancement module, a three-level RVQ codebook, and a multi-scale discriminator. The convergence speeds of these different modules are not uniform: the convolutional network can quickly learn the local waveform features of tactile signals, but the RVQ codebook requires many epochs to form a stable codeword distribution, and the frequency domain parameter kernel also needs to continuously learn the spectral patterns under different materials, sliding speeds, and visual motion intensities. Too few training epochs can easily lead to insufficient codebook utilization, coarse reconstructed waveforms, and loss of high-frequency textures; too many training epochs may cause the model to overfit the dataset containing only 280 original tactile signals, increasing training costs. Therefore, setting 500 epochs as the maximum training epochs can control the risk of overfitting while fully optimizing the codebook and perceptual frequency weights. Specifically, stable training of the network and codebook can be completed in the first 50 epochs, and complete end-to-end joint optimization can be performed from epochs 51 to 500, coupled with a validation set early stopping mechanism: training is terminated early when ST-SIM shows no improvement for 20 consecutive epochs. Therefore, 500 rounds is the upper limit of training rather than a fixed number of rounds that must be performed. This ensures that the model has enough convergence space while avoiding ineffective training based on the actual data.

[0031] This model can compress the original 44.8 kbps signal to 2.84 kbps, achieving a compression ratio of 15.8 and a bit rate reduction of 93.7%, while maintaining an ST-SIM of 0.915 and an end-to-end latency of less than 50 ms. It also utilizes the masking effect of visual motion on tactile perception to reduce the coding resources occupied by tactile frequency components that are difficult for the human eye to perceive when focusing on dynamic images. At the same time, it focuses on preserving tactile information in the human-sensitive frequency bands such as 150-200 Hz, thus balancing low bit rate, low latency, and realistic tactile perception.

[0032] S103: Calculate the residual signal based on the difference between the normalized reconstructed frame and the normalized tactile frame, perform perceptual weighting, dead zone processing and adaptive quantization on the residual signal, generate the residual quantization index, and entropy encode the RVQ codeword index and the residual quantization index based on preset auxiliary information to obtain the output bit stream, which is used as the tactile signal encoding result.

[0033] Calculate the residual signal e. The reconstructed haptic frame is obtained from the basic coding path. Then, the original haptic frame x is compared with the base reconstructed haptic frame. Subtracting them yields the residual signal: Where e represents the residual details not fully represented by the underlying coding path, and x represents the original haptic frame. This represents the basic reconstructed haptic frame; The logarithmic scale index of the residuals is extracted. To avoid strong residuals being truncated or weak but perceptually relevant residuals being lost due to a fixed quantization range, this invention first calculates the absolute peak value of the residuals in the current frame. After mapping it to the logarithmic field, it is uniformly quantized into an 8-bit scale index. : in This indicates a round-down operation. and This is a predefined safety boundary for the dynamic range. After obtaining the tick index, the encoder locally reconstructs the corresponding inverse-quantized tick s using the same inverse mapping as the decoder. This reconstructed tick s is not the original tick. It is used as an amplitude reference in the subsequent quantization stage, thereby ensuring consistency between the encoder and decoder.

[0034] The residual is subjected to perceptual frequency weighting. First, the residual signal e is transformed to the frequency domain, and then softening weights are constructed based on the obtained perceptual weight vector w. : After scale extraction, the residual signal e is reconstructed in the frequency domain, thus focusing coding resources on perceptually relevant components. The time-domain residual is transformed using a Fast Fourier Transform. A smoothing factor μ is introduced to construct a softened weight vector w to mitigate the problem of excessive attenuation of high-frequency residual components caused by directly using the original perceptual weight vector u in the main path, which leads to a smooth reconstruction result. Specifically, for B=3 and B=4, the empirical value of μ is set to 0.5; for B=5 and B=6, the empirical value of μ is set to 0.7. Stronger perceptual suppression is applied at lower bit rates to remove redundancy, while more residual spectral energy is retained at higher bit rates, thus preserving fine-grained tactile texture details.

[0035] To ensure that the residual retains the μ proportion of the original spectral energy even in frequency bands with low tactile sensitivity, the softened weights are applied to the residual spectrum, and the perceptually shaped residual is obtained through an inverse transform. : in and This represents the Fast Fourier Transform and the Inverse Fast Fourier Transform. This represents element-wise multiplication. Because... The magnitude of the residual after shaping will be limited by the reconstruction scale s, thus ensuring the numerical stability of the subsequent quantization process.

[0036] An adaptive quantizer with dead zone is used to discretize the perceptually shaped residual. First, the dead zone threshold τ is determined based on the relative threshold and the lower limit of absolute noise. The dead zone threshold τ is further determined by combining the relative threshold and the background absolute noise. in Indicates the dead zone ratio. This represents the absolute noise floor. Samples with amplitudes below τ will be set to zero. Amplitude quantization is performed on non-zero residual samples. The residual quantization bit width B can be 0, 3, 4, 5, or 6, where B=0 indicates that the residual branch is omitted and only the base code rate level is used. When residual quantization is enabled, B contains 1 symbol bit, and the number of amplitude quantization levels M is: Normalize the non-zero residual samples according to the inverse quantization scale s to obtain α= / s, and α∈[0,1], then the amplitude quantization index I is generated. mag : I mag =round(α·M) Where round(·) represents rounding, and the amplitude quantization index I is... mag Together with the symbol information, it serves as a residual index for subsequent entropy encoding.

[0037] The process of generating a layered bitstream includes: Based on the probability distribution of codewords and residual indices in the training haptic dataset, two independent Huffman codebooks are pre-constructed and shared with the receiving end during the initial connection establishment phase. Huffman lossless coding is performed on the residual vector quantization index, residual magnitude, and sign index respectively to eliminate statistical redundancy in discrete symbols; A final bitstream consisting of a frame header and a payload is generated. The frame header is of fixed length and is used to store auxiliary information required for decoding and reconstruction, including the normalized mean µ, standard deviation σ, and logarithmic scale index. And the residual quantization bit width B; the payload is of variable length and is used to store the residual vector quantization index and residual index after Huffman coding, and its length is determined by the basic quantization index length and the residual index length. The receiver parses the frame header and payload, recovers the residual vector quantization index and residual index according to the shared Huffman codebook, and then performs codeword lookup, latent representation reconstruction, decoder reconstruction, denormalization and residual compensation operations in sequence, finally outputting the reconstructed tactile signal.

[0038] Specifically, the present invention will be further illustrated below through embodiments: The experimental results below show that, compared with existing methods, the tactile signal encoding based on visual-tactile cross-sensory masking proposed in this invention utilizes the collaborative modulation of human physiological perception weights and learnable parameterized kernels to achieve adaptive selection and precise enhancement of the frequency domain features of vibration tactile signals, and achieves better generation results in noise suppression and detail restoration in key frequency bands.

[0039] To comprehensively and objectively evaluate the performance of the proposed solution compared to current cutting-edge methods, this invention conducted extensive comparative experiments based on the publicly available TUM standard vibration tactile signal dataset, and further verified the superior performance of the proposed solution in adjusting the bit rate and signal reconstruction and generation quality.

[0040] In this embodiment, the performance indicators used to evaluate the vibration tactile signal encoding scheme proposed in this invention are divided into three categories: signal-to-noise ratio, peak signal-to-noise ratio, and ST-SIM.

[0041] Signal-to-noise ratio (SNR): SNR is a signal quality evaluation metric. It measures the purity of a signal or the degree of noise interference by calculating the ratio of signal power to noise power (usually expressed in decibels). A higher SNR value generally indicates better signal quality and less noise interference.

[0042] Peak Signal-to-Noise Ratio (PSNR): PSNR is a signal quality evaluation metric that calculates the ratio of the energy of the peak signal to the average energy of the noise. It is the most common and widely used objective signal evaluation metric. A higher PSNR value indicates less distortion.

[0043] MSE is used to measure the average error between the original signal and the reconstructed signal, and is defined as: ST-SIM: ST-SIM is used to assess perceived quality. It shows a very high correlation with average subjective scores and can be considered a viable alternative to subjective testing. ST-SIM measures the original x and the reconstructed x. The formula for the perceived fidelity between them is as follows: Where S-SIMj and T-SIMj represent the spectral similarity and temporal similarity of the j-th block (block length 512), respectively. N b This indicates the total number of blocks contained in the signal.

[0044] Table 1. Comparison of delays for different tactile coding methods The method proposed in this invention has significant advantages, exhibiting better performance compared to four existing methods. It achieves efficient compression of tactile signals while maintaining sensory quality, resulting in lower buffer latency, faster encoding speed, and consistent tactile signal fidelity.

[0045] like Figure 2 As shown, Figure 2 This is a complete network structure diagram of the present invention. Stage 1: Preprocessing is located in the upper left corner of the diagram. The input raw triaxial tactile signal is first processed by frame segmentation. Subsequently, the segmented multidimensional tactile frames are spatially reduced using the DFT321 algorithm to output a one-dimensional tactile frame x. This one-dimensional frame x is then normalized by mean and standard deviation to finally generate a standardized one-dimensional temporal frame x' as the core input. Cross-sensory perception weight synthesis is located in the lower left corner of the diagram. This branch introduces a visual-tactile cross-sensory perception mechanism. First, the optical flow field of the video stream is calculated using a visual motion intensity extraction algorithm to obtain the visual motion velocity. Next, The input is fed into a pre-built visual-tactile perception threshold model, and the corresponding threshold-frequency curve is obtained. Finally, through reciprocal and normalization calculations, a perceptual weight curve capable of characterizing human physiological perception characteristics is synthesized. This weight will be synchronously transmitted to the frequency enhancement module in stage 2 and the frequency weighting module in stage 3. Stage 2: The PFE-RVQ model is located at the top of the diagram. After the normalized frame x' enters this model, it is first processed at the encoder end by the perception-driven frequency enhancement (PFE) module, combined with the perception weight. The frequency domain features are adaptively modulated and selected, then compressed and extracted into compact features through a one-dimensional convolutional layer. Next, a residual vector quantizer is used to perform multi-level vector quantization on these features, outputting a series of RVQ indices. At the decoder, the quantized features are inversely reconstructed through a transposed one-dimensional convolutional layer, a PFE module, and another one-dimensional convolutional layer to finally obtain the reconstructed frame. Original normalized frame x' and reconstructed frame The difference is taken, and the resulting residual signal e will be used as the input for the next stage. Stage 3: Residual quantization is located in the lower part of the figure. The residual signal e output from Stage 2 is processed in two ways: one way is used to extract the spatial scale factor through a logarithmic scaling extractor. The result is transformed into a logarithmic scale index, which is then dequantized to obtain the scaling factor s; another approach combines the sensing weights in the frequency domain to perform sensing frequency weighting, generating the sensing weighted residual. Finally, by using an adaptive quantizer combined with a scaling factor s and a preset quantization bit width, the quantization is... Uniform quantization is performed, and the final residual index is output. Stage 4: Bitstream packet generation is located on the right side of the diagram. At the encoding generation end, the RVQ index output from Stage 2 and the residual index output from Stage 3 are mapped to codebook 1 and codebook 2 respectively, and compressed efficiently using an entropy encoder. The compressed RVQ codewords and residual codewords serve as the payload of the data packet. Simultaneously, the mean and standard deviation, logarithmic scale index, and quantization bit width obtained in the preprocessing stage are combined and encapsulated into the packet header. Finally, the packet header and payload are assembled together into a standardized network data packet, generating the output bitstream and sending it to the receiving end.

[0046] like Figure 3 As shown, Figure 3 This is a schematic diagram of the perception-driven frequency enhancement module architecture of the present invention. The core design goal of this module is to adaptively modulate the frequency domain representation of the tactile signal by coordinating physiological perception weights and learnable parameterized kernels, thereby enhancing the fidelity of key perception frequency bands while suppressing irrelevant frequency band noise. The entire calculation process mainly consists of a feature frequency domain transformation backbone path and a perception control parameter generation branch. In the feature frequency domain transformation backbone path, the input time-domain normalized tactile frame x' is first mapped to a high-dimensional space through an embedding layer, and then transformed to the frequency domain using a fast Fourier transform to generate the corresponding frequency domain feature Q. Based on the physiological perception characteristics of humans, this frequency domain feature can be clearly divided into two sub-regions on the frequency axis: a perception-sensitive frequency band and a perception-insensitive frequency band. At the same time, the perception control parameter generation branch at the bottom runs in parallel to provide dynamic constraints for frequency domain modulation. This branch contains two sub-paths: the first path transforms the original continuous cross-sensory perception curve... Through interpolation, the signal is transformed into discrete perceptual weights W, which quantitatively characterize the physiological sensory sensitivity of human skin at different frequencies. The second path inputs a parameterized kernel R, initially uniformly and flatly distributed, into the kernel learning unit for network training, adaptively evolving it into a learnable parameterized kernel with nonlinear frequency selection characteristics. Subsequently, the frequency domain features Q in the main path are fused with the discrete perceptual weights W and the learnable parameterized kernel generated at the bottom through an element-wise weighted multiplication mechanism, thus constructing the modulated frequency domain features Y. In the spectral distribution of feature Y, background noise and redundant feature energy in the insensitive frequency band are significantly attenuated and suppressed, while key feature responses in the sensitive frequency band are well preserved and enhanced. Finally, the modulated frequency domain features Y are remapped back to the time domain through an inverse fast Fourier transform, ultimately outputting the reconstructed time-domain enhanced frame y. After processing by this module, the signal structure in both the time and frequency domains achieves a higher perceptual fit.

[0047] like Figure 4 As shown, Figure 4This is a comparison chart of the signal-to-noise ratio (SNR) of this invention with five other comparison methods. The horizontal axis of the chart represents the bit rate in kbps, and the vertical axis represents the signal-to-noise ratio (SNR) in dB. The legend is in the upper left corner. From top to bottom, the methods are VPC-DS (buffer = 512), PVC-SLP (buffer = 200), VC-PWQ (buffer = 512), RNVC (buffer = 512), RNVC (buffer = 64), and this invention (buffer = 64). Each point on the curve corresponding to this invention in the chart corresponds to the bit rate and SNR of this invention when the quantization bit width B = 3, 4, 5, and 6, respectively.

[0048] like Figure 5 As shown, Figure 5 This is a comparison chart of the peak signal-to-noise ratio (PSNR) of this invention with five other comparison methods. The horizontal axis of the chart represents the bit rate in kbps, and the vertical axis represents the peak signal-to-noise ratio (PSNR) in dB. The legend is in the upper left corner, and from top to bottom, the methods are VPC-DS (buffer = 512), PVC-SLP (buffer = 200), VC-PWQ (buffer = 512), RNVC (buffer = 512), RNVC (buffer = 64), and this invention (buffer = 64).

[0049] like Figure 6 As shown, Figure 6 This is a comparison chart of the present invention with five other ST-SIM methods. The horizontal axis represents the bit rate in kbps, the vertical axis represents ST-SIM, and the lower right corner is the legend. From top to bottom, the methods are VPC-DS (buffer = 512), PVC-SLP (buffer = 200), VC-PWQ (buffer = 512), RNVC (buffer = 512), RNVC (buffer = 64), and the present invention (buffer = 64).

[0050] like Figure 7 As shown, Figure 7 This is a flowchart of the workflow of the present invention. The TUM standard vibration tactile signal dataset is divided into two parts: a training set and a test set; the training set is used to train the PFE-RVQ model. The vibration tactile signals in the training set are first processed by window framing and the DFT321 algorithm, and then normalized to obtain normalized tactile frames.

[0051] In the PFE-RVQ model, on the one hand, visual motion intensity is extracted from synchronous dynamic visual information and combined with the visual-tactile perception threshold model, and frequency domain perception weight vector is generated after inverse and normalization processing; on the other hand, the normalized tactile frame is subjected to high-dimensional spatial projection and frequency domain transformation parameter optimization, and frequency domain perception weight is used for spectral weighting to enhance the perception-sensitive frequency components and suppress the insensitive frequency components under visual masking.

[0052] Subsequently, the spectrally weighted tactile features are sequentially fed into a texture one-dimensional convolutional encoder, a multi-level residual vector quantizer, a codebook reconstruction, a feature decoder, and an inverse normalization module to complete the encoding and reconstruction of the tactile signal. The trained PFE-RVQ model is used in the testing phase; the vibration tactile signals from the test set are input into the trained PFE-RVQ model, and further residual quantization and entropy encoding are performed to finally generate the output bitstream.

[0053] Example 2: To achieve the above objective, such as Figure 8 As shown, based on Embodiment 1, this invention discloses a tactile signal encoding system based on visual-tactile cross-sensory masking, comprising: The signal acquisition module 11 is used to acquire the original triaxial tactile signal and the corresponding video frame of the tactile signal, perform frame segmentation, triaxial dimensionality reduction and normalization processing on the original triaxial tactile signal to obtain the normalized tactile frame, and extract the corresponding video frame of the tactile signal to obtain the visual motion speed. Signal processing module 12 is used to input visual motion speed into a pre-established visual-touch cross-sensory perception threshold model and output frequency domain perception weights. It inputs normalized tactile frames and frequency domain perception weights into a perception-driven frequency enhancement and residual vector quantization model and outputs the difference between normalized reconstructed frames and normalized tactile frames and RVQ codeword indexes. The encoding output module 13 is used to calculate the residual signal based on the difference between the normalized reconstructed frame and the normalized tactile frame, perform perceptual weighting, dead zone processing and adaptive quantization on the residual signal, generate a residual quantization index, and entropy encode the RVQ codeword index and the residual quantization index based on preset auxiliary information to obtain the output bit stream as the tactile signal encoding result.

[0054] Based on the same inventive concept, this invention also provides a computer device, comprising: one or more processors, and a memory for storing one or more computer programs; the programs include program instructions, and the processor executes the program instructions stored in the memory. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, used to implement one or more instructions, specifically for loading and executing one or more instructions stored in a computer storage medium to implement the above-described method.

[0055] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, performs the above-described method. This storage medium can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0056] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0057] The foregoing has shown and described the basic principles, main features, and advantages of this disclosure. Those skilled in the art should understand that this disclosure is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of this disclosure. Various changes and modifications can be made to this disclosure without departing from its spirit and scope, and all such changes and modifications fall within the scope of this disclosure as claimed.

Claims

1. A tactile signal encoding method based on visual-tactile cross-sensory masking, characterized in that, The method includes the following steps: The original triaxial tactile signal and the corresponding video frame are obtained. The original triaxial tactile signal is then processed by framing, triaxial dimensionality reduction and normalization to obtain the normalized tactile frame. The visual motion speed is then extracted from the corresponding video frame. The visual motion speed is input into a pre-established visual-tactile cross-sensory perception threshold model, and the frequency domain perception weight is output. The normalized tactile frame and the frequency domain perception weight are input into the perception-driven frequency enhancement and residual vector quantization model, and the difference between the normalized reconstructed frame and the normalized tactile frame and the RVQ codeword index are output. The residual signal is calculated based on the difference between the normalized reconstructed frame and the normalized tactile frame. The residual signal is then subjected to perceptual weighting, dead zone processing, and adaptive quantization to generate a residual quantization index. The RVQ codeword index and the residual quantization index are then entropy encoded based on preset auxiliary information to obtain the output bit stream, which serves as the encoding result of the tactile signal.

2. The tactile signal encoding method based on visual-tactile cross-sensory masking according to claim 1, characterized in that, The triaxial dimensionality reduction process includes: The DFT321 dimensionality reduction algorithm is introduced to map the high-dimensional spatial signal in the original triaxial tactile signal into a one-dimensional signal that is easier to process. Specifically, according to the DFT321 dimensionality reduction algorithm, the spectrum... The expression for phase θ is as follows: Where f represents frequency, A1 represents the discrete Fourier transform of the original tactile frame along the X-axis, A2 represents the discrete Fourier transform of the original tactile frame along the Y-axis, and A3 represents the discrete Fourier transform of the original tactile frame along the Z-axis. The one-dimensional frame x can be derived from the preset spectrum and phase.

3. The tactile signal encoding method based on visual-tactile cross-sensory masking according to claim 1, characterized in that, The process of inputting visual motion velocity into a pre-established visual-tactile cross-sensory perception threshold model and outputting frequency domain perception weights includes: Threshold curve for a given visual motion speed Find the reciprocal to obtain the tactile sensitivity function. Then, the sensitivity function is normalized using its maximum value within the frequency range to obtain a perceptual weight with a value range of 0 to 1. The formula is as follows: Where ϵ is a regularization constant introduced to avoid division by zero. express The maximum value across the entire frequency range; After scale extraction, the residual signal e is reshaped in the frequency domain, thus focusing coding resources on the sensing-related components. The time-domain residual is transformed using a Fast Fourier Transform, and a smoothing factor μ is introduced to construct a softened weight vector w to mitigate the excessive attenuation of high-frequency residual components when directly using the original sensing weight vector u in the main path. The softened weights are constructed based on the obtained sensing weight vector w. As the frequency domain sensing weight, the formula is as follows: 。 4. The tactile signal encoding method based on visual-tactile cross-sensory masking according to claim 1, characterized in that, The processing steps for enhancing the perception-driven frequency and the residual vector quantization model include the following: For the obtained normalized haptic frame x, a data embedding layer consisting of one-dimensional convolutions will input a single channel. Projected into a higher-dimensional latent space, producing Where D represents the number of feature channels, and a Fast Fourier Transform is applied independently to each D channel to obtain... ; In the frequency domain, a learnable complex-valued parameter kernel is introduced and optimized during end-to-end training. By performing element-wise multiplication between Q and R, a learnable complex-valued parameter kernel R is introduced as an adaptive frequency domain equalizer to perform element-wise weighting on the spectral representation Q, resulting in a frequency-enhanced spectral representation: The kernel function R serves as an adaptive frequency domain equalizer. The normalized perceptual weight vector w is broadcast along the feature dimension to form a perceptual mask W, and then... Further perform perception weighting: By performing an inverse fast Fourier transform on the frequency domain feature Y, the time domain is restored and a frequency-enhanced frame y is obtained. Then, y is input into a depth encoder consisting of four cascaded one-dimensional convolutional layers to progressively compress the time dimension and generate a compact latent tactile representation z.

5. A tactile signal encoding method based on visual-tactile cross-sensory masking according to claim 4, characterized in that, The residual vector quantization process of the aforementioned sensing-driven frequency enhancement and residual vector quantization model includes the following steps: Residual vector quantization decomposes the original quantization task into L consecutive residual quantization stages, given the input features z and a set of independent codebooks. Each codebook contains K learnable codewords. The input to the l-th quantizer is the residual, where 1 ≤ l ≤ L, and its initial value is... ; The most recent codeword is selected from the current codebook, where represents the Euclidean norm. The residual is then updated and passed to the next stage. The final reconstructed feature ẑ is obtained by summing the codebook vectors selected in all L stages. The number of residual stages is set to L=3, and each codebook contains K=128 codewords. Encoding a single feature vector requires only three index values, totaling [number missing]. In the first training epoch, all codebooks are initialized using the K-means algorithm: Where K represents the number of discriminator models, and x' represents the normalized frame. Indicates a normalized reconstructed frame. This represents the output of the k-th discriminator in the last layer.

6. A tactile signal encoding method based on visual-tactile cross-sensory masking according to claim 5, characterized in that, The perception-driven frequency enhancement and residual vector quantization model is trained end-to-end, and the loss function used for training is as follows: in, The MSE loss is used to constrain the energy difference in the time domain between the normalized input frame and the normalized reconstructed frame. SSIM loss is used to maintain the consistency of the local temporal envelope and amplitude structure. The perception loss is used to increase the reconstruction weights of the perception-sensitive frequency bands based on the perception mask W. To combat the loss, a multi-scale discriminator is used to make the reconstructed tactile waveform more realistic. The feature matching loss is used to constrain the consistency of the intermediate feature maps of the discriminator. The loss commitment is used to constrain the encoder output to be close to the selected codeword.

7. A tactile signal encoding method based on visual-tactile cross-sensory masking according to claim 6, characterized in that, The MSE loss This item normalizes the input signal. With reconstructed signal To minimize the energy difference in the time domain, the MSE loss is defined as: Where N is the total number of samples in the signal; SSIM loss The loss term preserves the structural consistency of the local temporal envelope and amplitude contrast: Perceived loss By incorporating a predefined perceptual mask W, this loss term prompts the model to prioritize spectral reconstruction of perceptually sensitive frequency bands, while imposing a lower penalty on errors in the visual mask region: in This represents the complex coefficients of the k-th frequency band. It is a regularization constant introduced to prevent gradient vanishing in the mask region; Combating losses A multi-scale discriminator operating at a preset scale is employed to facilitate the generation of realistic tactile waveforms: in This represents the output of the discriminator at the j-th scale; Feature matching loss minimize The distance between the intermediate feature maps of the discriminator ensures that the generated signal remains close to the real signal in the representation space: in This represents the sum of the absolute differences of all elements in the feature map; Loss commitment Limiting the encoder output z to remain close to the selected codeword improves numerical stability during quantization. in This indicates that the gradient operator is stopped.

8. A tactile signal encoding method based on visual-tactile cross-sensory masking according to claim 1, characterized in that, The residual signal calculated based on the difference between the normalized reconstructed frame and the normalized haptic frame is as follows: Compare the original normalized haptic frame x with the base normalized reconstructed frame. Subtracting them yields the residual signal: Where 'e' represents the residual details that are not fully represented by the underlying coding path; Extract the logarithmic scale index of the residuals and calculate the absolute peak value of the residuals in the current frame. After mapping it to the logarithmic field, it is uniformly quantized into an 8-bit scale index. : in This indicates a round-down operation. and It is a predefined safety boundary with dynamic range; The residual signal e is transformed to the frequency domain, and a smoothing factor μ is introduced to construct a softened weight vector w. The softened weights are then constructed based on the obtained perceptual weight vector w. , The softened weights are applied to the residual spectrum, and the residual after perceptual shaping is obtained through inverse transform. : in and This represents the Fast Fourier Transform and the Inverse Fast Fourier Transform. This represents element-wise multiplication, because The magnitude of the residual after shaping will be limited by the reconstruction scale s; An adaptive quantizer with dead zone is used to discretize the perceptually shaped residual. First, the dead zone threshold τ is determined based on the relative threshold and the lower limit of absolute noise. The dead zone threshold τ is determined by combining the relative threshold and the background absolute noise. in Indicates the dead zone ratio. This represents the absolute noise floor; samples with amplitudes below τ will be set to zero. Amplitude quantization is performed on non-zero residual samples. The residual quantization bit width B can be 0, 3, 4, 5, or 6, where B=0 indicates that the residual branch is omitted and only the base code rate level is used. When residual quantization is enabled, B contains 1 symbol bit, and the number of amplitude quantization levels M is: Normalize the non-zero residual samples according to the inverse quantization scale s to obtain α= / s, and α∈[0,1], then the amplitude quantization index I is generated. mag : I mag =round(α·M) Where round(·) represents rounding, and the amplitude quantization index I is... mag Together with symbol information, they serve as residual indexes. The residual vector quantization index and residual quantization index are then losslessly entropy encoded and encapsulated into a layered bitstream suitable for network transmission.

9. A tactile signal encoding system based on visual-tactile cross-sensory masking, employing the tactile signal encoding method based on visual-tactile cross-sensory masking as described in any one of claims 1 to 8, characterized in that, include: The signal acquisition module is used to acquire the original triaxial tactile signal and the corresponding video frame of the tactile signal, perform frame segmentation, triaxial dimensionality reduction and normalization processing on the original triaxial tactile signal to obtain the normalized tactile frame, and extract the corresponding video frame of the tactile signal to obtain the visual motion speed. The signal processing module is used to input the visual motion speed into a pre-established visual-tactile cross-sensory perception threshold model and output the frequency domain perception weight. It inputs the normalized tactile frame and the frequency domain perception weight into the perception-driven frequency enhancement and residual vector quantization model and outputs the difference between the normalized reconstructed frame and the normalized tactile frame and the RVQ codeword index. The encoding output module is used to calculate the residual signal based on the difference between the normalized reconstructed frame and the normalized tactile frame, perform perceptual weighting, dead zone processing and adaptive quantization on the residual signal, generate a residual quantization index, and entropy encode the RVQ codeword index and the residual quantization index based on preset auxiliary information to obtain the output bit stream, which is used as the encoding result of the tactile signal.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, The memory stores a computer program that can run on a processor. When the processor loads and executes the computer program, it employs a tactile signal encoding method based on visual-tactile cross-sensory masking, as described in any one of claims 1 to 8.

Images