Multi-sensory interaction display system based on Internet

Abstract

The invention discloses a multi-sensory interaction display system based on the Internet, and the system comprises a multi-sensory data collection layer which carries out the synchronous collection of visual, auditory, touch and temperature data, carries out the preprocessing of the collected multi-modal data, and generates a multi-modal data package with a unified timestamp; the multi-modal data coding and transmission layer is used for compressing a multi-modal data packet based on a space-time joint coding algorithm and transmitting the multi-modal data packet through a low-delay transmission protocol; the terminal rendering and interaction layer is used for generating user interaction feedback in real time by adopting a dynamic multi-sensory fusion algorithm; and the edge-cloud cooperative computing layer is used for realizing low-delay feedback control and privacy protection based on a federated learning distributed computing framework. According to the invention, through layered decoupling and cross-layer optimization, strong space-time consistency of multi-modal interaction is realized, and through three innovative technologies of space-time joint coding, dynamic multi-sensory fusion and low-delay tactile control, high-fidelity and low-delay multi-sensory interaction is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of interactive display, and particularly relates to an Internet-based multi-sensory interactive display system. Background Art

[0002] With the improvement of technology, interactive display systems can be applied in different aspects, including virtual reality, human-computer interaction, remote operation and other fields. However, traditional interactive display systems mostly adopt single-sensory feedback technology. Existing independent interaction schemes based on vision or hearing have the problem of spatio-temporal fragmentation of multi-modal data, resulting in a tactile-visual synchronization error exceeding 50 ms and a fragmented user experience. The above problems seriously restrict the application effect of multi-sensory interactive display systems in scenarios such as virtual reality and remote operation. Summary of the Invention

[0003] Therefore, the present invention provides an Internet-based multi-sensory interactive display system to solve the problems in the prior art.

[0004] In order to achieve the above object, the present invention provides the following technical solutions:

[0005] An Internet-based multi-sensory interactive display system, comprising:

[0006] A multi-sensory data acquisition layer: synchronously acquire data of vision, audition, touch and temperature sense, preprocess the acquired multi-modal data, and then generate multi-modal data packets with unified timestamps;

[0007] A multi-modal data encoding and transmission layer: utilize the spatio-temporal correlation of multi-modal data to improve the compression efficiency, compress the multi-modal data packets based on a spatio-temporal joint encoding algorithm, and transmit them through a low-latency transmission protocol;

[0008] A terminal rendering and interaction layer: import the compressed multi-modal data packets, and adopt a dynamic multi-sensory fusion algorithm to generate real-time user interaction feedback, supporting tactile-visual-auditory-temperature sense synchronization;

[0009] An edge-cloud collaborative computing layer: based on a distributed computing framework of federated learning, dynamically allocate tasks to achieve low-latency feedback control and privacy protection; the cloud sends the trained federated learning model parameters to the edge nodes; the edge nodes feedback the local computing load status to the multi-modal data encoding and transmission layer, triggering dynamic adjustment of the encoding bit rate of the spatio-temporal joint encoding algorithm.

[0010] Further: The multi-modal data packets include RGB-D images, sound field impulse responses, tactile pressure matrices, and temperature field distributions.

[0011] Further: The preprocessing of the multi-modal data includes:

[0012] (1) Align in space and time, and use the extended Kalman filter to compensate for the physical position deviation of the sensor;

[0013] (2) Perform multi-modal fusion to construct a joint coordinate system.

[0014] Furthermore: The specific implementation steps of the spatio-temporal joint coding algorithm are as follows:

[0015] (1) Use the calibration matrix M st , and map the tactile coordinates to the visual coordinate system;

[0016] (2) Predict the tactile residual ΔH based on the convolutional neural network CNN;

[0017] ΔH = H - ∑w i,j V(x + i, y + j, t);

[0018] where, H is the target tactile signal or spatio-temporal coordinates, w i，j represents the weight coefficient; V(x + i, y + i, t) represents the predicted output of the convolutional neural network at the spatio-temporal coordinates (x + i, y + j, t);

[0019] (3) Perform wavelet transform and hierarchical entropy coding on the residual.

[0020] Furthermore: When training the weights of the spatio-temporal joint coding algorithm, it is necessary to first construct a cross-modal dataset and learn the visual-to-tactile mapping relationship through the convolutional neural network:

[0021]

[0022] where, min w represents optimizing the weight w to minimize the objective function; N is the total number of samples in the dataset; Hi represents the target value of the i-th data point; represents the double summation of the weights of the convolutional kernel; wj, l represents the weight of the convolutional kernel at the position (j, l); Vi(x + j, y + l) represents the input value of the i-th data point at the position (x + j, y + l); represents the square of the L2 norm; represents the L2 regularization term of the weight, and λ is the regularization parameter.

[0023] Furthermore: The implementation of the dynamic multi-sensory fusion algorithm includes the following steps:

[0024] (1) For data with transmission jitter exceeding ±5ms, compensate for the transmission jitter through the Kalman filter, and the state equation is defined as:

[0025]

[0026] where, is the state estimate at time t, A is the state transition matrix, is the state estimate at time t-1, K t is the Kalman gain matrix, Z t is the observed data at time t, H is the observation matrix; is the observation residual, also known as the innovation, representing the difference between the observed data Z t and the predicted observed value based on the prior estimate , which is used to correct the prior estimate to obtain a more accurate posterior estimate;

[0027] (2) Based on user annotation

[0028] (3) Weighted fusion of multimodal data, the weight W i The calculation formula is as follows:

[0029]

[0030] where, W i is the fusion weight of the i-th modality. After normalization, the sum of the weights of all modalities is 1; α is a hyperparameter for adjusting the weight distribution; S i is the score or confidence of the i-th modality; S j is the score of the j-th modality; represents the sum of the exponential scores of all modalities, which is used for normalization.

[0031] Furthermore: The low-latency feedback control is based on the adaptive pulse modulation of the piezoelectric array, and its control equation is:

[0032]

[0033] where, A i represents the tactile intensity coefficient of the i-th pulse signal; sin(2πf i t+φ i ) represents the sine wave coefficient of the i-th pulse signal, φ i represents the phase angle of the i-th pulse signal, 2πf i t represents the change of the sine wave with angular frequency 2πf i with time t, e -βt represents the factor that decays exponentially with time t, and β is the decay factor;

[0034] For the dynamic adjustment of parameters, where the tactile intensity Ai is determined by the user interaction pressure P, Ai = kp·P + b, where kp is the proportionality coefficient, which determines the influence degree of the user interaction pressure P on the tactile intensity Ai; b is the offset, representing the reference value of the tactile intensity Ai when the interaction pressure P is zero.

[0035] Further: The method for privacy protection mainly includes the following steps:

[0036] (1) Use Paillier homomorphic encryption for the local model gradient The edge node locally trains the tactile prediction model and uploads the encrypted gradient

[0037] (2) Add Gaussian noise during the aggregation process Meet (∈, δ)-differential privacy, where

[0038] (3) After cloud aggregation, distribute the global model w global .

[0039] Further: When the terminal rendering and interaction layer decodes the multimodal data packet, Reed-Solomon decoding is used for tactile data to recover lost packets; if the SSIM of visual data does not meet the standard, the edge node is requested to retransmit the key frame.

[0040] Still further: When the updated global model w global is input, optimize the PID parameters, and the data acquisition layer loads the new weight w global , update the residual calculation module in real time, and write the PID parameters into the FPGA of the temperature control module through the SPI protocol to achieve closed-loop regulation.

[0041] The present invention has the following advantages: Through hierarchical decoupling and cross-layer optimization, strong spatio-temporal consistency of multimodal interaction is achieved. Through three innovative technologies of spatio-temporal joint coding, dynamic multisensory fusion, and low-latency tactile control, high-fidelity and low-latency multisensory interaction is achieved.

[0042] Other features and advantages of the present invention will be described in the subsequent specification, and, in part, will be obvious from the specification or understood by implementing the present invention. Brief Description of the Drawings

[0043] To more intuitively illustrate the prior art and the present application, exemplary drawings are given below. It should be understood that the specific shapes and structures shown in the drawings generally should not be regarded as limiting conditions when implementing the present application; for example, those skilled in the art are capable of making conventional adjustments or further optimizations to the addition / deletion / attribution division of certain units (components), specific shapes, positional relationships, connection methods, dimensional proportional relationships, etc. based on the technical concept disclosed in the present application and the exemplary drawings.

[0044] Figure 1 It is a system block diagram of a multisensory interaction display system based on the Internet provided by the present application. Detailed Description of the Embodiments

[0045] The following specific embodiments illustrate the implementation manners of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all embodiments. It should be understood that these embodiments are only for further illustrating the present invention and cannot be construed as limiting the protection scope of the present invention. Technical engineers in this field can make some non-essential improvements and adjustments to the present invention according to the content of the above invention; based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the protection scope of the present invention.

[0046] Please refer to Figure 1 , a multi-sensory interactive display system based on the Internet, including:

[0047] Multi-sensory data acquisition layer: synchronously collect data of vision (RGB-D camera), audition (microphone array), touch (piezoelectric tactile feedback array), and temperature sense (thermoelectric module), preprocess the collected multi-modal data, and then generate multi-modal data packets with unified timestamps. The multi-modal data packets include RGB-D images, sound field impulse responses, tactile pressure matrices, and temperature field distributions.

[0048] The data preprocessing includes:

[0049] (1) Spatiotemporal alignment, using the extended Kalman filter (EKF) to compensate for the physical position deviation of the sensors;

[0050] (2) Multi-modal fusion, constructing a joint coordinate system (tactile pressure point → RGB-D pixel mapping).

[0051] Multi-modal data encoding and transmission layer: utilize the spatiotemporal correlation of multi-modal data to improve the compression efficiency, compress the multi-modal data packets based on the spatiotemporal joint coding algorithm (ST-JC), and transmit them through a low-latency transmission protocol.

[0052] The specific implementation steps of the spatiotemporal joint coding (ST-JC) algorithm are as follows:

[0053] (1) Use the calibration matrix M st , to map the tactile coordinates to the visual coordinate system;

[0054] (2) Predict the tactile residual ΔH based on the convolutional neural network CNN;

[0055] ΔH = H - ∑w i，j V(x + i, y + j, t);

[0056] Wherein, H is the target tactile signal or spatiotemporal coordinate, w i，jdenotes the weight coefficient; V(x+i, y+i, t) represents the predicted output of the convolutional neural network at the spatio-temporal coordinates (x+i, y+j, t).

[0057] (3) Perform wavelet transform and hierarchical entropy coding on the residuals.

[0058] During the training of the weights in the spatio-temporal joint coding algorithm (ST-JC), it is necessary to first train the dataset, that is, first construct a cross-modal dataset (RGB-D image + tactile pressure matrix), and learn the visual-to-tactile mapping relationship through a convolutional neural network (CNN):

[0059]

[0060] where min w represents optimizing the weight w to minimize the objective function; N is the total number of samples in the dataset; Hi represents the target value of the i-th data point; represents the double summation of the weights of the convolutional kernel; wj, l represents the weight of the convolutional kernel at the position (j, l); Vi(x+j, y+l) represents the input value of the i-th data point at the position (x+j, y+l); represents the square of the L2 norm; represents the L2 regularization term of the weight, and λ is the regularization parameter.

[0061] Terminal rendering and interaction layer: Import the compressed multi-modal data packet, and use the dynamic multi-sensory fusion algorithm (DMF) to generate real-time user interaction feedback, supporting tactile-visual-auditory-thermal sensation synchronization.

[0062] For the decoding of the data packet, the tactile data uses Reed-Solomon decoding to recover the lost packets; if the SSIM of the visual data does not meet the standard, request the edge node to retransmit the key frame (I frame);

[0063] The implementation of the dynamic multi-sensory fusion algorithm includes the following steps:

[0064] (1) For data with a transmission jitter exceeding ±5 ms, compensate the transmission jitter through Kalman filtering, and the state equation is defined as:

[0065]

[0066] where is the state estimate at time t, A is the state transition matrix, is the state estimate at time t-1, K t is the Kalman gain matrix, Z t is the observed data at time t, and H is the observation matrix. is the observation residual, also known as the innovation, representing the observed data Zt The difference from the predicted observation value based on the prior estimate is used to correct the prior estimate and obtain a more accurate posterior estimate.

[0067] For example, if the predicted position observation value Z t = [12, 18] T , and the Kalman gain K t = 0.6, then after correction, the state estimate value is:

[0068]

[0069] (2) Generate the Gaussian weight W vision (x, y);

[0070] (3) Weighted fusion of multi-modal data, with the weight W i The calculation formula is as follows:

[0071]

[0072] where W i is the fusion weight of the i-th modality. After normalization, the sum of the weights of all modalities is 1; α is a hyperparameter for adjusting the weight distribution; S i is the score or confidence of the i-th modality; S j is the score of the j-th modality; represents the sum of the exponential scores of all modalities, which is used for normalization.

[0073] Edge-cloud collaborative computing layer: A distributed computing framework based on federated learning that dynamically allocates tasks to achieve low-latency inference and privacy protection.

[0074] The cloud (running the Swin Transformer model in the cloud) sends the trained federated learning model parameters (such as the tactile-temperature mapping model) to the edge nodes; the edge nodes (deploying a lightweight LSTM model at the edge nodes) feedback the local computing load status to the encoding layer, triggering dynamic adjustment of the ST-JC coding rate.

[0075] The federated learning mechanism includes:

[0076] (1) Local training, where the edge nodes update the tactile model using differential privacy (DP-SGD algorithm, noise scale σ = 0.1).

[0077] (2) Global aggregation, where the cloud adopts the FedAvg++ algorithm (weighted average + gradient clipping) to prevent model divergence.

[0078] In addition, low-latency haptic feedback control, based on adaptive pulse modulation (APM) of a piezoelectric array, supports microsecond-level response; its control equation is:

[0079]

[0080] where A i represents the haptic intensity coefficient of the i-th pulse signal; sin(2πf i t + φ i ) represents the sine wave coefficient of the i-th pulse signal, φ i represents the phase angle of the i-th pulse signal, 2πf i t represents the change of a sine wave with an angular frequency of 2πf i over time t, and e -βt represents a factor that decays exponentially over time t, and β is the decay factor.

[0081] Parameter dynamic adjustment: The haptic intensity Ai is determined by the user interaction pressure P, Ai = kp·P + b, where kp is the proportionality coefficient that determines the influence degree of the user interaction pressure P on the haptic intensity Ai; b is the offset, representing the reference value of the haptic intensity Ai when the interaction pressure P is zero.

[0082] The main steps of the privacy protection method include:

[0083] (1) Use Paillier homomorphic encryption for the local model gradient The edge node locally trains the haptic prediction model and uploads the encrypted gradient

[0084] (2) Add Gaussian noise during the aggregation process to satisfy (∈, δ)-DP (differential privacy), where

[0085] (3) After cloud aggregation, distribute the global model w global .

[0086] In this embodiment, when the updated global model w global is input, the PID parameters are optimized, the data acquisition layer loads the new weight w global , the residual calculation module is updated in real time, and the PID parameters are written into the FPGA of the temperature control module through the SPI protocol to achieve closed-loop regulation.

[0087] In this embodiment, the processing method for sensitive data is that when face feature extraction is only completed at the edge, the original image data is immediately discarded.

[0088] In addition, in this embodiment, when the haptic feedback delay exceeds 8 ms, the QUIC protocol is triggered to switch to the low-latency mode, that is, the congestion window is halved.

[0089] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. An Internet-based multi-sensory interactive display system, characterized in that Including: Multi-sensory data acquisition layer: synchronously acquire visual, auditory, tactile, and temperature data, preprocess the acquired multi-modal data, and then generate multi-modal data packets with unified timestamps; Multi-modal data encoding and transmission layer: utilize the spatio-temporal correlation of multi-modal data, compress multi-modal data packets based on spatio-temporal joint encoding algorithms, and transmit them through low-latency transmission protocols; Terminal rendering and interaction layer: import the compressed multi-modal data packets, and use dynamic multi-sensory fusion algorithms to generate real-time user interaction feedback, supporting tactile-visual-auditory-temperature synchronization; Edge-cloud collaborative computing layer: based on the distributed computing framework of federated learning, dynamically allocate tasks to achieve low-latency feedback control and privacy protection; the cloud sends the trained federated learning model parameters to the edge nodes; the edge nodes feedback the local computing load status to the multi-modal data encoding and transmission layer, triggering the dynamic adjustment of the encoding bit rate of the spatio-temporal joint encoding algorithm.

2. The multisensory interactive display system based on the Internet according to claim 1, characterized in that The multi-modal data packets include RGB-D images, sound field impulse responses, tactile pressure matrices, and temperature field distributions.

3. An Internet-based multi-sensory interactive display system according to claim 1, characterized in that, The preprocessing of the multi-modal data includes: (1) Spatio-temporal alignment, using the extended Kalman filter to compensate for the physical position deviation of the sensors; (2) Multi-modal fusion, constructing a joint coordinate system.

4. A multi-sensory interactive display system based on the Internet according to claim 1, characterized in that The specific implementation steps of the spatio-temporal joint encoding algorithm are as follows: (1) Use the calibration matrix M st to map the tactile coordinates to the visual coordinate system; (2) Predict the tactile residual ΔH based on the convolutional neural network CNN; ΔH = H - ∑w i,j V(x + i, y + j, t); where H is the target tactile signal or spatio-temporal coordinates, and w i，j represents the weight coefficient; V(x+i, y+i, t) represents the predicted output of the convolutional neural network at the spatio-temporal coordinates (x+i, y+j, t); (3) Perform wavelet transform and hierarchical entropy encoding on the residual.

5. An Internet-based multi-sensory interactive display system according to claim 4, characterized in that, When training the weights of the spatio-temporal joint encoding algorithm, it is necessary to first construct a cross-modal dataset and learn the mapping relationship from vision to touch through a convolutional neural network: where, min w represents optimizing the weight w to minimize the objective function; N is the total number of samples in the dataset; Hi represents the target value of the i-th data point; represents the double summation of the weights of the convolutional kernel; wj, l represents the weight of the convolutional kernel at position (j, l); Vi(x + j, y + l) represents the input value of the i-th data point at position (x + j, y + l); represents the square of the L2 norm; represents the L2 regularization term of the weight, and λ is the regularization parameter.

6. The multisensory interaction display system based on the Internet according to claim 1, characterized in that, The implementation of the dynamic multi-sensory fusion algorithm includes the following steps: (1) For data with transmission jitter exceeding ±5ms, compensate for the transmission jitter through the Kalman filter, and the state equation is defined as: Among them, is the state estimate value at time t, A is the state transition matrix, is the state estimate value at time t - 1, K t is the Kalman gain matrix, Z t is the observed data at time t, H is the observation matrix; is the observation residual, also known as the innovation, representing the difference between the observed data Z t and the predicted observed value based on the prior estimate to correct the prior estimate and obtain a more accurate posterior estimate; (2) Based on user notes (3) Weighted fusion of multimodal data, with weight W i The calculation formula is as follows: Among them, W i is the fusion weight of the i-th modality. After normalization, the sum of the weights of all modalities is 1; α is a hyperparameter for adjusting the weight distribution; S i is the score or confidence of the i-th modality; S j is the score of the j-th modality; represents the summation of the exponential scores of all modalities for normalization.

7. An Internet-based multi-sensory interactive display system according to claim 1, characterized in that, The low-latency feedback control is based on the adaptive pulse modulation of the piezoelectric array, and its control equation is: Among them, A i represents the tactile intensity coefficient of the i-th pulse signal; sin(2πf i t + φ i ) represents the sine wave coefficient of the i-th pulse signal, φ i represents the phase angle of the i-th pulse signal, 2πf i t represents the change of the sine wave with angular frequency 2πf i with time t, e -βt represents a factor that decays exponentially with time t, and β is the decay factor; For the dynamic adjustment of parameters, where the tactile intensity Ai is determined by the user interaction pressure P, Ai = kp·P + b, where kp is the proportionality coefficient, which determines the influence degree of the user interaction pressure P on the tactile intensity Ai; b is the offset, indicating the reference value of the tactile intensity Ai when the interaction pressure P is zero.

8. A multi-sensory interactive display system based on the Internet according to claim 1, characterized in that, The method of privacy protection mainly includes the following steps: (1) Use Paillier homomorphic encryption for local model gradients Edge nodes locally train the tactile prediction model and upload the encrypted gradients (2) Add Gaussian noise during the aggregation process Satisfy (∈, δ)-differential privacy, where (3) After cloud aggregation, the global model w is sent down global .

9. A multi-sensory interactive display system based on the Internet according to any one of claims 1-8, characterized in that When the terminal rendering and interaction layer decodes the multi-modal data packets, the tactile data uses Reed-Solomon decoding to recover the lost packets; if the SSIM of the visual data does not meet the standard, request the edge node to retransmit the key frames.

10. A multi-sensory interactive display system based on the Internet according to any one of claims 1-8, characterized in that, When the updated global model w is input global the PID parameters are optimized, and the new weights w are loaded in the data acquisition layer global the residual calculation module is updated in real time, and the PID parameters are written into the temperature control module FPGA through the SPI protocol to achieve closed-loop regulation.

Images