Multi-Sensory Interaction Synchronization System and Method Based on Dynamic Timestamps

Abstract

The present invention discloses a multi-sensory interaction synchronization system and method based on dynamic timestamps, which relates to the technical field of human-computer interaction. The system includes: a time reference module for providing a unified time reference for all devices through a high-precision time source; a signal encoding module for encoding visual, auditory, tactile, and olfactory signals into data frames in a unified format and embedding dynamic timestamps in the data frames, where the dynamic timestamps are used to indicate the decoding time of the signals at the terminal device. The present invention realizes millisecond-level synchronization of multi-modal signals through a high-precision time reference, dynamic timestamps, and dynamic compensation technology, enhances the immersion feeling, reduces the device integration cost by means of cross-platform protocol adaptation and semantic web technology, strengthens data privacy and system robustness by integrating federated learning and edge computing, and adaptively optimizes the interaction experience in complex scenarios by combining dynamic priority scheduling and semantic web technology, providing core technical support for multiple fields.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction, and particularly to a multi-sensory interaction synchronization system and method based on dynamic timestamps. Background Art

[0002] In immersive interaction scenarios such as the metaverse and virtual reality, the precise synchronization of multi-modal sensory signals is the core foundation for constructing a realistic experience. However, traditional multi-sensory interaction systems face severe challenges: due to significant differences in hardware processing cycles, communication protocols (Bluetooth / Wi-Fi / wired), and data processing logics among different types of devices (such as XR headsets, haptic gloves, and odor generators), the transmission and processing delays of visual, auditory, tactile, and olfactory signals can vary by more than 50 ms. For example, the Bluetooth transmission delay of haptic gloves is typically 20 - 30 ms, while the wired transmission delay of XR headsets is only 5 - 10 ms. This delay difference can cause "tactile-visual asynchrony" - when a user touches an object in a virtual scene, the tactile feedback lags significantly behind the visual image, creating a strong sense of sensory disconnection.

[0003] More critically, existing technologies rely on static calibration methods (presetting fixed delay compensation values) and cannot adapt to the real-time fluctuations in delays (with a fluctuation range of up to ±20 ms) caused by network fluctuations, device load changes, or multi-person interactions in dynamic scenarios. When the scene complexity suddenly increases (such as rapid virtual environment switching or multi-device concurrent interaction), the static compensation mechanism fails, and the synchronization error further expands, resulting in discomfort such as dizziness and operation disconnection for users, severely restricting the popularization of immersive experiences. In addition, the closed interfaces and inconsistent protocols of device manufacturers lead to extremely high costs for multi-modal data fusion, further exacerbating the system integration difficulty. Summary of the Invention

[0004] The purpose of the present invention is to provide a multi-sensory interaction synchronization system and method based on dynamic timestamps to solve the problems raised in the above background art.

[0005] To achieve the above purpose, the present invention provides the following technical solution: A multi-sensory interaction synchronization system based on dynamic timestamps, comprising:

[0006] A time reference module for providing a unified time reference for all devices through a high-precision time source, such that the clock synchronization error of the devices does not exceed 1 μs, and the high-precision time source includes an atomic clock or a satellite time service device;

[0007] A signal encoding module for encoding visual, auditory, tactile, and olfactory signals into data frames in a unified format and embedding dynamic timestamps in the data frames, where the dynamic timestamps are used to indicate the decoding time of the signals at the terminal devices;

[0008] A delay compensation module, which is used to dynamically adjust the signal transmission timing based on a preset sensory signal priority, where signals with a higher priority are preferentially allocated transmission resources;

[0009] An interface adaptation module, which is used to support multiple communication protocols and achieve the rapid access and data interaction of devices from different manufacturers.

[0010] Preferably, the calculation formula of the dynamic timestamp is: where is the reference time obtained by the time reference module, accurate to the nanosecond level; is the transmission delay between devices predicted based on a long short-term memory network, with the unit of millisecond; is the scene complexity coefficient, which is obtained by normalizing parameters related to the scene, and the value range is from 0 to 1; is the dynamic compensation factor, and its calculation formula is where is the empirical coefficient, is a very small constant to prevent division by zero.

[0011] Preferably, the preset order of sensory signal priorities is:

[0012] The priority of tactile signals is higher than that of visual signals, the priority of visual signals is higher than that of olfactory signals, and the priority of olfactory signals is higher than that of auditory signals;

[0013] Signals with different priorities are transmitted using different transmission protocols. In the signal buffer of the terminal device, signals with a higher priority have the priority decoding permission.

[0014] Preferably, the time reference module synchronizes the clocks of all devices through the Network Time Synchronization Protocol, and the Network Time Synchronization Protocol includes the Network Time Protocol or the Precision Time Protocol;

[0015] Among them, in an environment lacking a high-precision time source, the time reference module can achieve distributed time synchronization through a network composed of edge computing nodes and through the Reference Broadcast Synchronization Protocol, and the time synchronization error does not exceed 5 μs.

[0016] Preferably, the delay compensation module includes:

[0017] A prediction unit, which predicts the transmission delay through a long short-term memory network model integrating an attention mechanism, and the calculation formula is where is the historical hidden state, is the attention weight matrix at each time step, and this matrix dynamically adjusts the weight coefficient according to the signal type, is a multi-dimensional input vector containing historical latency data, network status parameters, and device load; or, model training is performed through federated learning technology, and each device calculates the model gradient locally and uploads it to the server for aggregation and update to avoid the transmission of raw latency data;

[0018] A scheduling unit, which adjusts the signal priority in real time through a fuzzy logic algorithm, combining the scene type and the sensory signal fusion threshold. The priority adjustment formula is , where is the signal type, is the scene type, is the sensory signal fusion threshold; or, by accessing the electroencephalogram signal feedback path, the signal compensation priority is dynamically adjusted according to the electroencephalogram signal monitoring results.

[0019] Preferably, the interface adaptation module includes:

[0020] A protocol conversion sub-module for implementing the conversion of multiple communication protocols, which include but are not limited to OpenXR, DCPv2.0, and MQTT.

[0021] The present invention also provides a multi-sensory interaction synchronization method based on dynamic timestamps, including the following steps:

[0022] Synchronize the clocks of all devices through a high-precision time source to generate a unified reference time;

[0023] Encode visual, auditory, tactile, and olfactory signals into data frames in a unified format, predict the transmission latency between devices based on a long short-term memory network, calculate a dynamic compensation factor in combination with the scene complexity coefficient, generate a dynamic timestamp, and embed it into the data frame;

[0024] Dynamically adjust the signal transmission timing according to the preset sensory signal priority, and preferentially allocate transmission resources to signals with high priority;

[0025] The terminal device decodes the signal according to the dynamic timestamp and feeds back the actual latency data to the latency compensation module to form an accuracy synchronization closed loop.

[0026] The present invention also provides an electronic device, which is a physical device. The electronic device includes:

[0027] A processor and a memory, and the memory is communicatively connected to the processor;

[0028] The memory is used to store at least one executable instruction executed by the processor, and the processor is used to execute the executable instruction to implement the multi-sensory interaction synchronization method based on dynamic timestamps as described above.

[0029] The present invention also provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the multi-sensory interaction synchronization method based on a dynamic timestamp as described above is implemented.

[0030] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0031] By adopting an atomic clock or satellite time service in combination with the Network Time Protocol, the present invention constructs a globally unified time coordinate system for multi-modal signals, eliminates clock drift of devices, ensures that the time deviation across devices is at an extremely low level, realizes high-precision real-time synchronization by uniformly encoding multi-sensory signals and embedding dynamic timestamps, meets the needs of human sensory fusion, predicts transmission delays through an LSTM neural network to provide a reliable basis for signal synchronization, guarantees the transmission of key signals and improves interaction fluency by setting the priority of sensory signals and reasonably allocating resources, realizes device compatibility and reduces integration costs by supporting multiple communication protocols, further optimizes prediction accuracy and protects data privacy by integrating attention mechanisms, federated learning, etc., and enhances system adaptability and reliability by combining dynamic priority scheduling and semantic web technology, comprehensively improving the interaction experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 FIG. is a schematic structural diagram of a multi-sensory interaction synchronization system based on a dynamic timestamp provided by an embodiment of the present invention;

[0033] Figure 2 FIG. is an example flowchart of a multi-sensory interaction synchronization method based on a dynamic timestamp provided by an embodiment of the present invention;

[0034] Figure 3 FIG. is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0035] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0036] Please refer to Figure 1 , this embodiment provides a multi-sensory interaction synchronization system based on a dynamic timestamp, including:

[0037] A time reference module 11, configured to provide a unified time reference for all devices through a high-precision time source, so that the clock synchronization error of the devices does not exceed 1 μs, and the high-precision time source includes an atomic clock or a satellite time service device;

[0038] A signal encoding module 12, configured to encode visual, auditory, tactile, and olfactory signals into data frames in a unified format, and embed a dynamic timestamp in the data frames, where the dynamic timestamp is used to indicate the decoding time of the signals at the terminal device;

[0039] A delay compensation module 13, configured to dynamically adjust the signal transmission timing based on a preset priority of sensory signals, where signals with higher priority are preferentially allocated transmission resources;

[0040] An interface adaptation module 14, configured to support multiple communication protocols and enable rapid access and data interaction of devices from different manufacturers.

[0041] In an optional embodiment, the time reference module 11 synchronizes the clocks of all devices through a network time synchronization protocol, and the network time synchronization protocol includes the Network Time Protocol or the Precision Time Protocol;

[0042] Wherein, in an environment lacking a high-precision time source, the time reference module 11 can implement distributed time synchronization through a network composed of edge computing nodes through the Reference Broadcast Synchronization Protocol, and the time synchronization error does not exceed 5 μs.

[0043] Specifically, the time reference module 11 uses an atomic clock (such as a rubidium atomic clock with a time accuracy of ≤1 μs) or satellite time service (GPS / Beidou) as the reference time source, and synchronizes the clocks of all devices through the Network Time Protocol (NTP) or the Precision Time Protocol (PTP) to ensure that the cross-device time deviation is ≤1 μs.

[0044] The system constructs a reference clock system using a high-precision time source, and the core solution includes two types of high-precision time service technologies. First, a rubidium atomic clock is selected as the local time reference, which realizes timing based on the energy level transition characteristics of rubidium atoms, and the time accuracy can reach the sub-microsecond level (≤1 μs), which is suitable for a closed environment with low network dependence; second, access to the global satellite navigation system (such as GPS, Beidou), and realize wide-area time synchronization through the nanosecond-level accurate time signals broadcast by satellites. At the clock synchronization protocol level, support the hybrid deployment of the Network Time Protocol (NTP) and the Precision Time Protocol (PTP): The NTP protocol realizes millisecond-level fast synchronization through UDP packets, which is suitable for devices with low real-time requirements; the PTP protocol is based on the IEEE1588 standard, adopts the hardware timestamp technology, and through the master-slave clock architecture and the path delay compensation algorithm, can strictly control the cross-device time deviation within 1 μs to ensure the high unity of the clocks of multi-sensory devices.

[0045] This time synchronization mechanism constructs a unified time coordinate system covering the entire system, fundamentally solving the problem of time inconsistency in the process of multi-modal signal acquisition and processing. The cumulative error of the device's local clock caused by factors such as crystal oscillator temperature drift and aging will lead to misalignment of multi-modal data on the time axis (such as different delays between audio and video signals). Through real-time calibration and dynamic compensation, the global time reference can control the time deviation of each device below the sensory threshold, ensuring that multi-modal signals such as vision, hearing, and touch are fused and processed in the same time dimension, providing a basic guarantee for the accurate analysis and collaborative response of subsequent multi-sensory interaction data.

[0046] In an offline environment without atomic clocks or satellite signals (such as underground industrial scenarios), distributed time synchronization is achieved through the networking of edge computing nodes. By adopting an improved RBS (Reference Broadcast Synchronization) protocol, the edge nodes calibrate time through multi-hop broadcasts, with an error ≤ 5 μs; it forms a'main-backup collaboration' with the atomic clock synchronization mode to enhance the robustness of the system.

[0047] In an optional embodiment, the calculation formula for the dynamic timestamp is: , where is the reference time obtained through the time reference module 11, accurate to the nanosecond level; is the transmission delay between devices predicted based on a long short-term memory network, with the unit of millisecond; is the scene complexity coefficient, obtained by normalizing parameters related to the scene, with a value range of 0 to 1; is the dynamic compensation factor.

[0048] Specifically, in a multi-sensory interaction system, the differences between different modal signals are extremely large. To achieve precise synchronization, signals such as vision (RGB frames), hearing (audio PCM data), touch (pressure / vibration signals), and smell (gas release instructions) need to be unified and processed into a standardized data frame format. This format mainly consists of the following two parts:

[0049] Payload: The payload part carries the original sensory data, which are the core information carriers of system interaction. For visual signals, it appears as a pixel matrix, recording the color and brightness information of each frame; auditory signals exist in the form of audio sampling points, with each sampling point corresponding to the amplitude of the sound waveform at a specific moment; touch signals include the pressure values collected by pressure sensors or the drive parameters of vibration motors; and smell signals are reflected as the command parameters for controlling gas release devices, such as the type, concentration, and duration of the released gas.

[0050] Dynamic Timestamp: The dynamic timestamp module is the key to achieving precise synchronization of multimodal signals. Its core lies in embedding the decoding time of the signal expected by the terminal device.

[0051] In the above formula for the dynamic timestamp:

[0052] The UTC timestamp synchronized by the atomic clock is used. The UTC timestamp synchronized by the atomic clock is accurate to the nanosecond level, providing an absolute time reference for the entire system and ensuring the consistency of the time scales between different devices;

[0053] A prediction model is constructed based on the long short-term memory network (LSTM). This model dynamically predicts the transmission delay of the current signal between different devices (in milliseconds, ms) by learning historical transmission data (such as features like packet size, network bandwidth changes, device distance, etc.). For example, when there is network congestion or the devices are far apart, the value will increase accordingly;

[0054] is used to dynamically adjust the synchronization accuracy according to the scene complexity. Its calculation formula is:

[0055]

[0056] Where is the empirical coefficient, with a default value of 0.8, which can be fine-tuned according to the actual application scenario;

[0057] is a very small constant to prevent division by zero, with a value of ;

[0058] is the scene complexity coefficient, with a value range between 0 and 1, obtained by normalizing key parameters such as frame rate, number of devices, and network load. For example, in a high-frame-rate game scenario or a complex environment with a large number of devices working together, the value approaches 1, and the system will enhance the synchronization accuracy compensation to ensure the synchronization quality of multimodal signals.

[0059] In an optional embodiment, the preset priority order of the sensory signals is:

[0060] The priority of the tactile signal is higher than that of the visual signal, the priority of the visual signal is higher than that of the olfactory signal, and the priority of the olfactory signal is higher than that of the auditory signal;

[0061] Signals with different priorities are transmitted using different transmission protocols. In the signal buffer of the terminal device, signals with higher priority have the priority decoding permission.

[0062] Optionally, a hierarchical scheduling strategy is constructed based on the physiological characteristics of human senses, and the specific priority settings are as follows:

[0063] Tactile signal: As the fastest perception channel of the human body, the nerve conduction speed is about 100 m / s, and the human perception threshold is strictly controlled at ≤10 ms. To meet the ultra-low latency requirement, the UDP protocol combined with FEC (Forward Error Correction) coding is used for transmission, and a dedicated priority queue is configured in the terminal device to ensure the priority decoding and rendering of tactile data.

[0064] Visual signal: The visual nerve conduction speed is about 40 m / s, and the perception threshold is ≤50 ms. The system adopts a dynamic bitrate adjustment strategy, automatically switching between H.264 / H.265 coding standards in combination with the network condition, while ensuring the image quality and balancing the transmission latency.

[0065] Olfactory signal: The olfactory conduction speed is the slowest, about 1 m / s, and the perception threshold is ≤500 ms. Given its high requirement for transmission reliability, the TCP protocol is used for transmission, and a circular buffer is set at the receiving end, and smooth playback is achieved through a double-buffering mechanism.

[0066] Auditory signal: Although the sound wave conduction speed reaches 340 m / s, the brain has a relatively high tolerance for audio synchronization. The system adopts an adaptive resampling technology, dynamically adjusting the playback speed by calculating the PTS (Presentation Time Stamp) difference between the audio and the video to ensure the final synchronization of multi-modal signals.

[0067] Based on the characteristic differences of the above various sensory signals, the system adopts a priority scheduling strategy: high-priority signals (such as tactile signals) are preferentially allocated network bandwidth and have the priority decoding right in the terminal device buffer to ensure the immediacy of key sensory feedback; while low-priority signals (such as olfactory signals) reasonably utilize network resources on the basis of ensuring data reliability to achieve the efficient coordination and precise synchronization of multi-sensory signals.

[0068] In an optional embodiment, the delay compensation module 13 includes:

[0069] A prediction unit, which predicts the transmission delay through a long short-term memory network model integrating an attention mechanism, and the calculation formula is , where is the historical hidden state, is a multi-dimensional input vector containing historical delay data, network state parameters, and device load, is the attention weight matrix for each time step. This matrix dynamically adjusts the weight coefficients according to the signal type. Specifically, the system assigns corresponding weight coefficients based on the characteristics of different sensory signals and their contribution to the synchronization accuracy: for tactile signals with high time-delay sensitivity, the weight coefficient is set to 1.5 to enhance their priority in the synchronization calculation; for auditory signals, considering their relatively low requirement for time accuracy, the weight coefficient is set to 0.8. Through this dynamic adjustment mechanism, the model can adaptively balance the fusion strategy of multi-modal data. Through actual measurement verification, the prediction error can be stably controlled within 1.5 ms, significantly improving the synchronization accuracy of the system.

[0070] Alternatively, model training is carried out through federated learning technology. Each device calculates the model gradient locally and uploads it to the server for aggregation and update to avoid the transmission of original delay data.

[0071] Under the model training mechanism through federated learning technology, after each terminal device completes the calculation of the model gradient locally, only the encrypted gradient parameters are uploaded to the central server for aggregation and update, avoiding the cross-device transmission of original delay data throughout the process. This training mode of "data does not move while the model moves" can, while ensuring data privacy and security, make full use of the computing power resources of edge devices, effectively solve the problem of data leakage risk existing in traditional centralized training, and achieve the dual goals of privacy protection and model performance optimization.

[0072] A scheduling unit, which adjusts the signal priority in real time through a fuzzy logic algorithm, combining the scene type and the sensory signal fusion threshold. The priority adjustment formula is , where is the signal type, is the scene type, is the sensory signal fusion threshold; or, by accessing the electroencephalogram signal feedback path, the signal compensation priority is dynamically adjusted according to the electroencephalogram signal monitoring results.

[0073] Specifically, in a multi-modal interaction scenario, the dynamic adjustment mechanism of signal priority can be precisely optimized according to the specific application scenario. For example, in a spatial movie experience, the system recognizes the immersive movie-watching demand based on the scene type parameters, and temporarily increases the priority weight coefficient of the olfactory signal from the default 0.3 to 0.8 through a dynamic adjustment algorithm, forming a synergy with the 0.85 weight of the visual signal, so that when the audience feels the interstellar explosion scene, the smoke and burnt smell can be triggered synchronously, significantly enhancing the immersive experience of the plot.

[0074] In addition, the system supports the feedback closed-loop access of electroencephalogram (EEG) signals, and the amplitude and latency of the P300 EEG component are monitored in real time through a high-precision head-mounted device. When the characteristics of decreased immersion are detected (such as the P300 wave amplitude being lower than 20% of the baseline value and lasting for more than 3 seconds), an adaptive compensation mechanism is triggered: first, the signal short board of the current scene is judged through a Bayesian inference model. If the auditory signal is missing, the priority of the audio signal is increased by 30%, and the fusion thresholds of other modalities are adjusted synchronously, forming an intelligent control closed-loop of "physiological signal acquisition - state evaluation - parameter optimization - multi-modal compensation" to achieve the dynamic optimization of immersion with millisecond-level response.

[0075] In an optional embodiment, the interface adaptation module 14 includes:

[0076] A protocol conversion sub-module for implementing the conversion of multiple communication protocols, where the communication protocols include but are not limited to OpenXR, DCPv2.0, and MQTT;

[0077] A device description parsing sub-module. The device description parsing sub-module uses semantic web technology to describe device attributes using ontology language, and parses device attributes through a semantic inference engine to achieve the automatic matching of device capabilities and the reasonable allocation of compensation resources; or, through the distributed time synchronization function between edge computing nodes, the time synchronization ability of the system in special environments is enhanced.

[0078] Optionally, in the device description parsing sub-module, an OWL ontology language is used to construct a device capability description framework, and the semantic expression of device characteristics is realized through the structured definition of classes, attributes, and instances. Taking a haptic feedback device as an example, by defining a subclass HapticGlove of the Device class and setting the forceFeedbackAccuracy attribute to 0.1mm; for an odor generating device, it is accurately characterized through the OdorGenerator class and the responseTime attribute (with a value of 100ms). Combining semantic inference engines such as Jena, a semantic matching algorithm is automatically executed based on the device capability description, and a dynamic compensation model is established for the capability differences between heterogeneous devices. When the system detects the access of a device with high latency but high precision, the inference engine will dynamically allocate compensation resources such as the GPU acceleration module and low-latency network channels in the cloud computing resource pool to the target device according to preset rules to ensure the real-time performance and accuracy of multi-modal data synchronization processing.

[0079] Alternatively, in the edge computing scenario without atomic clock deployment, the ReferenceBroadcastSynchronization (RBS) protocol is deeply optimized, introducing an adaptive weight adjustment mechanism and a two-way timestamp interaction strategy. By constructing an edge node topology awareness model, multi-hop path analysis is performed on the nodes participating in synchronization, and nodes with excellent link quality and fewer hops are preferentially selected as reference nodes. During the synchronization process, each node periodically broadcasts a beacon packet containing the local timestamp, and the receiving node fuses the timestamps received multiple times through the Kalman filtering algorithm to effectively suppress clock drift. Through actual measurement verification, in a typical industrial Internet of Things scenario, the time synchronization error of this solution can be stably controlled within 5 μs. At the same time, the system designs the atomic clock mode as the main time source, which automatically switches when an external high-precision time source is available, forming a primary and backup redundant architecture with the distributed synchronization mechanism to ensure the reliability of time synchronization in a complex network environment.

[0080] In this embodiment, the present invention constructs a globally unified time coordinate system for multi-modal signals by using an atomic clock or satellite time service in combination with the Network Time Protocol, eliminates the clock drift of devices, ensures that the time deviation across devices is at an extremely low level, realizes high-precision real-time synchronization by uniformly encoding multi-sensory signals and embedding dynamic timestamps, meets the requirements of human sensory fusion, predicts the transmission delay through the LSTM neural network to provide a reliable basis for signal synchronization, guarantees the transmission of key signals and improves the interaction fluency by setting the priority of sensory signals and reasonably allocating resources, achieves device compatibility and reduces the integration cost by supporting multiple communication protocols, further optimizes the prediction accuracy and protects data privacy by integrating attention mechanisms, federated learning, etc., and enhances the system adaptability and reliability by combining dynamic priority scheduling and semantic web technology, comprehensively improving the interaction experience.

[0081] Based on the above embodiments, as Figure 2 shown, this embodiment further provides a multi-sensory interaction synchronization method based on dynamic timestamps, including the following steps:

[0082] Step 100, synchronize the clocks of all devices through a high-precision time source to generate a unified reference time.

[0083] Specifically, a high-precision atomic clock or a global satellite navigation system (such as GPS, Beidou) is deployed as the time source, and the Network Time Protocol (NTP) or the Precision Time Protocol (PTP) is used to synchronize and calibrate the clocks of all devices participating in the interaction. Specifically, the master device periodically sends a time synchronization packet to the slave device, and the slave device calculates the time offset and compensates it after receiving it. After multiple iterations, a unified reference time is generated to ensure that the time error within the system is controlled within the microsecond level.

[0084] Step 200: Encode visual, auditory, tactile, and olfactory signals into data frames in a unified format. Predict the transmission delay between devices based on a long short-term memory network, calculate a dynamic compensation factor by combining with the scene complexity coefficient, and generate a dynamic timestamp and embed it into the data frame.

[0085] Specifically, adopt cross-modal data fusion technology to encode signals such as vision (RGB video stream, depth image), audition (multi-channel audio), touch (force feedback data), and olfaction (gas concentration parameter encoding) into data frames in a unified format that complies with the ISO / IEC 23008-11 standard;

[0086] Build a transmission delay prediction model based on a long short-term memory network (LSTM), and predict the transmission delay between devices by analyzing historical transmission data (including features such as network bandwidth and device load). ;

[0087] Introduce a scene complexity coefficient , which is weighted and calculated from parameters such as the number of dynamic objects in the scene and the density of interaction devices, and its value range is [0, 1];

[0088] According to the formula generate a dynamic timestamp, where is an adjustable compensation coefficient, and its value is determined by experimental optimization. Finally, embed the timestamp into the data frame header.

[0089] Step 300: Dynamically adjust the signal transmission timing according to the preset sensory signal priorities, and preferentially allocate transmission resources to signals with higher priorities.

[0090] Specifically, establish a sensory signal priority matrix, divide the signals into emergency categories (such as tactile feedback, warning audio), important categories (such as key action visual signals), and ordinary categories (such as environmental background sounds, non-critical visual elements). Based on the dynamic programming algorithm, combined with the current network state and device resource occupancy, adopt a priority queue scheduling strategy for signals with higher priorities, and preferentially allocate transmission resources by reserving bandwidth, compressing low-priority data, etc., to ensure that the transmission delay of key signals is lower than the threshold.

[0091] Step 400: The terminal device decodes the signal according to the dynamic timestamp and feeds back the actual delay data to the delay compensation module to form an accuracy synchronization closed loop.

[0092] Specifically, after receiving the data frame, the terminal device extracts the dynamic timestamp , calculates the actual transmission delay in combination with the local time, takes the difference between the actual transmission delay and the predicted delay as feedback data, updates the LSTM prediction model parameters through the backpropagation algorithm, and dynamically adjusts the compensation factor , a closed-loop synchronization mechanism of "prediction - transmission - feedback - optimization" is formed to continuously improve the synchronization accuracy of multi-sensory signals.

[0093] In this embodiment, the present invention constructs a globally unified time coordinate system for multi-modal signals by using an atomic clock or satellite time service combined with the Network Time Protocol, eliminates device clock drift, ensures that the cross-device time deviation is at an extremely low level, realizes high-precision real-time synchronization by uniformly encoding multi-sensory signals and embedding dynamic timestamps, meets the needs of human sensory fusion, predicts transmission delays through an LSTM neural network to provide a reliable basis for signal synchronization, ensures the transmission of key signals by setting the priority of sensory signals and reasonably allocating resources, improves interaction fluency, realizes device compatibility by supporting multiple communication protocols, reduces integration costs, further optimizes prediction accuracy and protects data privacy by integrating attention mechanisms, federated learning, etc., and enhances system adaptability and reliability by combining dynamic priority scheduling and semantic web technology, comprehensively improving the interaction experience.

[0094] Furthermore, the cache data processing device for intelligently analyzing the operation habits of investment users can run the above multi-sensory interaction synchronization system and method based on dynamic timestamps. For specific implementation, reference can be made to the method embodiment, which will not be elaborated here.

[0095] On the basis of the above embodiment, as Figure 3 shown, this embodiment further provides an electronic device, which includes:

[0096] At least one processor 22, at least one memory 21, a communication interface 23, and a communication bus 24, and the processor 22 is communicatively connected to the memory 21;

[0097] In this embodiment, the memory 21 can be implemented in any suitable manner. For example, the memory 21 can be a read-only memory, a mechanical hard disk, a solid-state drive, or a USB flash drive, etc.; the memory 21 is used to store executable instructions executed by at least one of the processors.

[0098] In this embodiment, the processor 22 can be implemented in any suitable manner. For example, the processor 22 can take the form of, for example, a microprocessor or a processor, and a computer-readable medium storing computer-readable program code (such as software or firmware) executable by the (micro)processor, logic gates, switches, an Application-Specific Integrated Circuit (ASIC), a programmable logic controller, and an embedded microcontroller, etc.; the processor is used to execute the executable instructions to implement the multi-sensory interaction synchronization method based on dynamic timestamps as described above.

[0099] Based on the above embodiments, the present invention further provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, it implements the multi-sensory interaction synchronization method based on dynamic timestamps as described above.

[0100] Those of ordinary skill in the art can realize that the modules and method steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0101] Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the above-described devices, equipment, and modules can refer to the corresponding processes in the foregoing method embodiments, and will not be repeated herein.

[0102] In the several embodiments provided in the present application, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division, and there can be other division methods in actual implementation. For example, multiple modules or units can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or equipment can be in an electrical, mechanical, or other forms.

[0103] The modules described as separate components may or may not be physically separated. The components shown as modules may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0104] In addition, the functional modules in each embodiment of the present invention can be integrated into one processing module, or each module can exist physically alone, or two or more modules can be integrated into one module.

[0105] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage media include: U disk, mobile hard disk, read-only storage server, random access storage server, disk or optical disk, and other media that can store program instructions.

[0106] In addition, it should be noted that the combination of the various technical features in this case is not limited to the combination described in the claims of this case or the combination described in the specific embodiments. All technical features described in this case can be freely combined or combined in any way unless there is a contradiction between them.

[0107] It should be noted that the above examples are only specific embodiments of the present invention, and the present invention is obviously not limited to the above examples, and there are many similar variations. All variations directly derived or associated from the contents disclosed by the technicians in this field should fall within the protection scope of the present invention.

[0108] The above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multi-sensory interaction synchronization system based on dynamic timestamps, characterized in that Including: A time reference module, which is used to provide a unified time reference for all devices through a high-precision time source, so that the clock synchronization error of the devices does not exceed 1 μs. The high-precision time source includes an atomic clock or a satellite time service device; A signal encoding module, which is used to encode visual, auditory, tactile, and olfactory signals into data frames in a unified format, and embed a dynamic timestamp in the data frame. The dynamic timestamp is used to indicate the decoding time of the signal at the terminal device; A delay compensation module, which is used to dynamically adjust the signal transmission timing based on a preset sensory signal priority, where signals with higher priority are preferentially allocated transmission resources; An interface adaptation module, which is used to support multiple communication protocols to achieve fast access and data interaction of devices from different manufacturers; Among them, the calculation formula of the dynamic timestamp is as follows: , where is the reference time obtained by the time reference module, accurate to the nanosecond level; is the transmission delay between devices predicted based on the long short-term memory network, with the unit of millisecond; is the scene complexity coefficient, which is obtained by normalizing the parameters related to the scene, and the value range is from 0 to 1; is the dynamic compensation factor, and its calculation formula is , where is the empirical coefficient, is a very small constant to prevent division by zero; The delay compensation module includes: A prediction unit, which predicts transmission delay through a long short-term memory network model integrating an attention mechanism, and the calculation formula is , where is the historical hidden state, is the attention weight matrix at each time step, and this matrix dynamically adjusts the weight coefficient according to the signal type, is a multi-dimensional input vector containing historical delay data, network state parameters, and device load; or, model training is performed through federated learning technology, and each device calculates the model gradient locally and uploads it to the server for aggregation and update to avoid the transmission of original delay data; A scheduling unit, which adjusts the signal priority in real time through a fuzzy logic algorithm, combining the scene type and the sensory signal fusion threshold. The priority adjustment formula is , where is the signal type, is the sensory signal fusion threshold; or, by accessing the electroencephalogram signal feedback path, dynamically adjusting the signal compensation priority according to the electroencephalogram signal monitoring results.

2. The multi-sensory interaction synchronization system based on dynamic timestamps according to claim 1, characterized in that, The preset order of sensory signal priorities is: The priority of tactile signals is higher than that of visual signals, the priority of visual signals is higher than that of olfactory signals, and the priority of olfactory signals is higher than that of auditory signals; Signals with different priorities are transmitted using different transmission protocols. In the signal buffer of the terminal device, signals with higher priority have the priority decoding permission.

3. The multi-sensory interaction synchronization system based on dynamic timestamps according to claim 1, wherein The time reference module synchronizes the clocks of all devices through the Network Time Synchronization Protocol, and the Network Time Synchronization Protocol includes the Network Time Protocol or the Precision Time Protocol; Among them, in an environment lacking a high-precision time source, the time reference module can achieve distributed time synchronization through a network composed of edge computing nodes through the Reference Broadcast Synchronization Protocol, and the time synchronization error does not exceed 5 μs.

4. The multi-sensory interaction synchronization system based on dynamic timestamps according to claim 1, wherein The interface adaptation module includes: A protocol conversion sub-module, which is used to implement the conversion of multiple communication protocols. The communication protocols include OpenXR, DCPv2.0, and MQTT; A device description parsing sub-module. The device description parsing sub-module uses semantic web technology to describe device attributes using ontology language, and parses device attributes through a semantic reasoning engine to achieve automatic matching of device capabilities and reasonable allocation of compensation resources; or, enhance the time synchronization ability of the system in a special environment through the distributed time synchronization function between edge computing nodes.

5. A multi-sensory interaction synchronization method based on dynamic timestamps, characterized in that, Including the following steps: Synchronize the clocks of all devices through a high-precision time source to generate a unified reference time; Encode visual, auditory, tactile, and olfactory signals into data frames in a unified format, predict the transmission delay between devices based on a long short-term memory network, calculate a dynamic compensation factor in combination with the scene complexity coefficient, generate a dynamic timestamp and embed it into the data frame; Dynamically adjust the signal transmission timing according to the preset sensory signal priority, and preferentially allocate transmission resources to signals with higher priority; The terminal device decodes the signal according to the dynamic timestamp and feeds the actual delay data back to the delay compensation module to form a precision synchronization closed loop; Among them, the calculation formula of the dynamic timestamp is as follows: , where is the reference time obtained by the time reference module, accurate to the nanosecond level; is the transmission delay between devices predicted based on the long short-term memory network, with the unit of millisecond; is the scene complexity coefficient, which is obtained by normalizing the parameters related to the scene, and the value range is from 0 to 1; is the dynamic compensation factor, and its calculation formula is , where is the empirical coefficient, is a very small constant to prevent division by zero; The dynamically adjusting the signal transmission timing according to the preset sensory signal priority and preferentially allocating transmission resources to signals with higher priority includes: Transmission delay prediction is performed using a long short-term memory network model integrated with an attention mechanism, and the calculation formula is , where is the historical hidden state, is the attention weight matrix for each time step, and this matrix dynamically adjusts the weight coefficients according to the signal type, is a multi-dimensional input vector containing historical delay data, network state parameters, and device load; or, model training is performed through federated learning technology, and each device calculates the model gradient locally and uploads it to the server for aggregation and update to avoid the transmission of original delay data; Through the fuzzy logic algorithm, the signal priority is adjusted in real time by combining the scene type and the sensory signal fusion threshold. The priority adjustment formula is , where is the signal type, is the scene type, is the sensory signal fusion threshold; or, by accessing the electroencephalogram signal feedback path, the signal compensation priority is dynamically adjusted according to the electroencephalogram signal monitoring results.

6. An electronic device, characterized in that, The electronic device includes: A processor and a memory, and the memory is communicatively connected to the processor; The memory is used to store at least one executable instruction executed by the processor, and the processor is used to execute the executable instruction to implement the multi-sensory interaction synchronization method based on a dynamic timestamp according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, it implements the multi-sensory interaction synchronization method based on a dynamic timestamp according to any one of claims 1 to 4.

Images