A VR panorama glasses data synchronization and transmission method based on multi-sensor fusion

Abstract

This invention discloses a VR panoramic glasses data synchronization and transmission method based on multi-sensor fusion, relating to the fields of computer networks and multimedia real-time transmission technology. This invention utilizes extended Kalman filtering to perform time alignment and unified modeling of various sensor data, including eye-tracking, inertial measurement, gestures, and environmental data. It encodes the user's head position, posture, gaze direction, and gestures into a single user behavior vector. This allows the subsequent system to avoid dealing with raw sensor streams of varying formats, instead making decisions based on a clearly structured behavior space, thus significantly reducing the cumulative errors of posture jitter and gaze deviation. Furthermore, a long short-term memory network is introduced to perform time-series modeling of the behavior vector and environmental context, predicting interactive hotspot areas in the virtual scene and mapping the hotspot distribution to priority weights for different data streams. This ensures that sensor streams related to the vicinity of the gaze and rapid movements can still achieve a high service level even under bandwidth constraints.

Description

Technical Field

[0001] This invention relates to the field of computer network and multimedia real-time transmission technology, and in particular to a method for data synchronization and transmission of VR panoramic glasses based on multi-sensor fusion. Background Technology

[0002] In virtual collaborative meetings, users communicate via voice, gestures, and eye guidance in a shared virtual space using head-mounted display devices (HMDs). The system typically needs to transmit simultaneously: rendered frames or panoramic / volume media, spatial audio, and multi-source sensor data such as head posture, gestures, eye tracking, and controller / IMU data. To reduce end-to-end latency, the industry often adopts architectures such as point-to-point / mesh, selective forwarding units (SFUs), multipoint control units (MCUs), and cloud / edge rendering, combined with adaptive bitrate, scalable coding (layered subscription), congestion control, FEC / NACK, multipath transmission, and edge computing offloading to adapt to bandwidth fluctuations and terminal heterogeneity.

[0003] However, in multi-user, high-concurrency scenarios, video streams consume large amounts of bandwidth, while sensor and audio streams, although with smaller bandwidths, are highly sensitive to latency. Existing cross-stream scheduling cannot reliably guarantee low-latency transmission of critical control / sensor data. Furthermore, local clocks on different devices drift, media and sensor sampling rates vary, and timestamp alignment and cross-source synchronization mechanisms are insufficient, easily leading to lag and jitter between motion-driven virtual avatars and audio / video. At the same time, the complexity of end-to-end links causes latency accumulation: encoding / decoding, queuing, packet loss recovery, cross-domain routing, and multi-hop forwarding are superimposed, causing inconsistent interactive responses, and in severe cases, audio-visual desynchronization or session interruption. Under mechanisms such as SFU / layered subscription, the coupling between server-side forwarding decisions and client-side adaptive strategies is insufficient, making it difficult to achieve stable fairness and consistent experience at the group level.

[0004] Therefore, a synchronization and transmission method with unified time base and cross-stream priority control for multi-sensor fusion is still needed to reduce latency and jitter in critical data paths under high concurrency conditions and improve the consistency and stability of multi-user collaboration. Summary of the Invention

[0005] In view of the aforementioned existing problems, the present invention is proposed.

[0006] This invention provides a VR panoramic glasses data synchronization and transmission method based on multi-sensor fusion to solve the problems of asynchronous data from multiple sensor sources and severe bandwidth competition in existing multi-person VR collaboration, which leads to prominent issues such as delays in key interactive streams and audio-visual asynchrony.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0008] This invention provides a method for data synchronization and transmission of VR panoramic glasses based on multi-sensor fusion, applied to a virtual collaborative system including VR panoramic glasses and edge computing nodes. The VR panoramic glasses include multiple sensors, including:

[0009] S1, Collect multi-sensor data output by the multiple sensors, wherein the multiple sensors include at least an eye-tracking sensor, an inertial measurement unit sensor, a gesture sensor, and an environmental sensor;

[0010] S2, the multi-sensor data is fused to generate user behavior data representing user posture, gaze direction and gestures;

[0011] S3, determine the transmission priority of various data streams based on the user behavior data;

[0012] S4, perform multi-path transmission scheduling on at least two physical or logical transmission paths according to the transmission priority, and allocate high-priority data streams and low-priority data streams to different transmission paths respectively.

[0013] S5, perform synchronization processing on the multiple data streams obtained through the multi-path transmission to align the playback timing of the multiple data streams.

[0014] As a preferred embodiment of the VR panoramic glasses data synchronization and transmission method based on multi-sensor fusion described in this invention, the fusion of the multi-sensor data includes:

[0015] Based on the extended Kalman filter algorithm, the state prediction and update of the multi-sensor data output by the eye-tracking sensor, inertial measurement unit sensor, gesture sensor and environmental sensor are performed to generate a unified user behavior vector, and the user behavior data includes the user behavior vector;

[0016] When fusing multi-sensor data output from the eye-tracking sensor, inertial measurement unit sensor, gesture sensor, and environmental sensor:

[0017] The system aligns multi-sensor data along the time axis and constructs a user behavior vector that simultaneously represents the user's head spatial position, head linear velocity, head posture, gaze direction, and gesture. In the state prediction stage, the user behavior vector from the previous moment and the output of the inertial measurement unit (IMU) sensor are used as inputs. A nonlinear state transition model based on head kinematics, gaze dynamics, and gesture dynamics is used for state extrapolation, and the prediction uncertainty is modeled by incorporating process noise covariance. In the state update stage, a joint observation model is constructed based on the measurement mechanisms of eye-tracking sensors, IMU sensors, gesture sensors, and environmental sensors. The predicted state is mapped to the observation space of each sensor. The predicted state is corrected through innovative extended Kalman filtering, Kalman gain, and covariance update steps, resulting in a unified user behavior vector updated by multi-sensor fusion, which is output as part of the user behavior data.

[0018] As a preferred embodiment of the VR panoramic glasses data synchronization and transmission method based on multi-sensor fusion described in this invention, the method includes: determining the transmission priority of various data streams based on the user behavior data, which includes:

[0019] The user behavior data is used as an input sequence. A long short-term memory neural network model is used to predict the user's interaction hotspots in the virtual scene, and the transmission priority weights for different sensor data streams are calculated based on the interaction hotspots.

[0020] Inputting the user behavior data into a long short-term memory neural network model includes:

[0021] User behavior vectors within a predetermined time window are collected on the time axis and concatenated with contextual features obtained from environmental sensors and virtual scene grid location encoding to form the input sequence of a Long Short-Term Memory (LSTM) neural network. The LSM neural network is then used to perform temporal modeling on the input sequence to obtain the probability distribution of interaction hotspots in multiple spatial regions divided within the virtual scene. For each type of sensor data stream, a correlation parameter between the sensor and each spatial region is pre-defined. Based on the interaction hotspot probability distribution and the corresponding correlation parameter, the unnormalized priority score of the sensor data stream is calculated. Through interval compression and normalization, the unnormalized priority scores of various sensor data streams are converted into comparable transmission priority weights at the same scale. During the model training phase, a joint loss function is used, including a cross-entropy term based on interaction region annotation, a prior distribution constraint term based on user gaze generation, and a time smoothing term that penalizes changes in the hotspot probability distribution over consecutive time intervals, to jointly optimize the LSM neural network model and the correlation parameters.

[0022] As a preferred embodiment of the VR panoramic glasses data synchronization and transmission method based on multi-sensor fusion described in this invention, the environmental sensors include a temperature sensor and a light sensor, and the method further includes:

[0023] Based on the environmental information collected by the temperature sensor and the light sensor, the transmit power, coding rate and retransmission strategy in the multipath transmission scheduling are adjusted to obtain a data transmission strategy that meets the current environmental conditions.

[0024] As a preferred embodiment of the VR panoramic glasses data synchronization and transmission method based on multi-sensor fusion described in this invention, the multi-path transmission scheduling includes:

[0025] The first transmission path carrying high-priority data streams is configured as an ultra-reliable low-latency communication channel, and the second transmission path carrying low-priority data streams is configured as a high-capacity wireless channel.

[0026] As a preferred embodiment of the VR panoramic glasses data synchronization and transmission method based on multi-sensor fusion described in this invention, the synchronization processing of the multiple data streams obtained through the multi-path transmission includes:

[0027] Assign logical clock timestamps to data streams from multiple VR panoramic glasses;

[0028] The multiple data streams are sorted and buffered according to the logical clock timestamp;

[0029] The playback timing of each data stream is adjusted based on the sorting results to achieve synchronization of multi-user data streams.

[0030] As a preferred embodiment of the VR panoramic glasses data synchronization and transmission method based on multi-sensor fusion described in this invention, the method is at least partially executed on the edge computing node, which is used to perform the fusion processing of the multi-sensor data and the determination of the transmission priority.

[0031] As a preferred embodiment of the VR panoramic glasses data synchronization and transmission method based on multi-sensor fusion described in this invention, the method further includes:

[0032] The transmitted data will be fragmented into multiple data blocks and sent.

[0033] When network packet loss is detected, the incomplete data blocks from the plurality of data blocks are input into a pre-trained generative adversarial network model, which reconstructs the lost data blocks and prioritizes the recovery of data blocks carrying high-priority data streams during reconstruction.

[0034] As a preferred embodiment of the VR panoramic glasses data synchronization and transmission method based on multi-sensor fusion described in this invention, the method includes: multi-path transmission scheduling of at least two physical or logical transmission paths according to the transmission priority, which includes:

[0035] Different lengths of send buffer queues are allocated to data streams with different transmission priorities. The length of the send buffer queue corresponding to a high-priority data stream is less than the length of the send buffer queue corresponding to a low-priority data stream.

[0036] As a preferred embodiment of the VR panoramic glasses data synchronization and transmission method based on multi-sensor fusion described in this invention, the method further includes:

[0037] Based on the link delay, packet loss rate, and available bandwidth measured during the multipath transmission process, the transmission priority weight or the multipath transmission scheduling strategy is periodically updated.

[0038] The beneficial effects of this invention are as follows: By constructing a unified user behavior data path and behavior perception transmission scheduling mechanism between VR panoramic glasses and edge computing nodes, this invention effectively alleviates the interaction instability problems caused by asynchronous multi-source sensors, network bandwidth competition, and end-to-end latency fluctuations in the prior art.

[0039] This invention utilizes extended Kalman filtering to perform time alignment and unified modeling of data from various sensors, including eye-tracking, inertial measurement, gestures, and the environment. It encodes the user's head position, posture, gaze direction, and gestures into a single user behavior vector. This allows subsequent systems to make decisions based on a clearly structured behavior space, rather than dealing with raw sensor streams of varying formats, thus significantly reducing the cumulative errors of posture jitter and gaze deviation. The invention introduces a long short-term memory network to perform time-series modeling of behavior vectors and environmental context, predicting interactive hotspots in the virtual scene and mapping their distribution to priority weights for different data streams. This ensures that sensor streams near the gaze and those related to rapid actions receive a high service level even under bandwidth constraints, while weakly correlated or less important areas are automatically downgraded, mitigating the problem of critical data being overwhelmed by treating all streams equally. On the transmission side, multi-path scheduling allocates high-priority data streams to ultra-reliable low-latency channels and low-priority data streams to high-capacity channels. Combined with different lengths of transmission buffer queues and link quality feedback, it prioritizes the latency and jitter metrics of critical interactive streams during network fluctuations, resulting in more consistent and timely action responses perceived by the user. Meanwhile, logical clock timestamps are used to sort and buffer multi-user, multi-channel data, aligning the playback timing of each data stream, reducing the relative misalignment between virtual avatar actions and audio / video between users, and improving timing consistency in collaborative scenarios. Furthermore, performing multi-sensor fusion and priority calculation at edge nodes can shorten the decision path and reduce the burden on the terminal, making it possible to introduce complex models; in the event of packet loss, generative adversarial networks are used to prioritize the reconstruction of high-priority data blocks, further reducing the disruption of image continuity and key action reproduction caused by packet loss.

[0040] Based on the above characteristics, this method, without relying on a specific network protocol, collaboratively optimizes the data synchronization and transmission strategy in VR multi-user collaboration from the perception layer, decision layer, and transmission layer. To a certain extent, it overcomes the latency fluctuations caused by bandwidth competition and the synchronization errors of multi-source sensors in the background technology, and improves the stability and consistency of the immersive collaborative experience. Attached Figure Description

[0041] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation on the scope of this application.

[0042] Figure 1 This is a flowchart illustrating the VR panoramic glasses data synchronization and transmission method based on multi-sensor fusion in the embodiment. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0044] All terms used in this application (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0045] For example, the terms “first” and “second” used in this application are only used to distinguish and describe similar objects, to differentiate the first object from another object, and are not used to describe a specific order or sequence, nor should they be interpreted as indicating or implying relative importance.

[0046] This application proposes a data synchronization and transmission method for VR panoramic glasses based on multi-sensor fusion, applied to a virtual collaborative system including VR panoramic glasses and edge computing nodes. The VR panoramic glasses include multiple sensors, combined with... Figure 1 As shown, the method includes:

[0047] S1, collects multi-sensor data output from multiple sensors, including at least an eye-tracking sensor, an inertial measurement unit sensor, a gesture sensor, and an environmental sensor;

[0048] S2, fuses multi-sensor data to generate user behavior data that characterizes user posture, gaze direction, and gestures;

[0049] S3 determines the transmission priority of various data streams based on user behavior data;

[0050] S4, perform multi-path transmission scheduling on at least two physical or logical transmission paths according to transmission priority, and allocate high-priority data streams and low-priority data streams to different transmission paths respectively.

[0051] S5 performs synchronous processing on the multiple data streams obtained through multi-path transmission, aligning the playback timing of the multiple data streams.

[0052] In this embodiment, multi-sensor data and multiple data streams can interact between the VR panoramic glasses and edge computing nodes via existing real-time transmission protocols. Eye-tracking, inertial measurement, and gesture data are typically generated by the device at a fixed sampling period, while environmental sensor data can be collected at a lower frequency. The sampling period can be set according to the terminal hardware capabilities and target interaction latency. For example, the sampling frequency of the inertial measurement unit can be selected from 100 to 1000 times per second, the sampling frequency of the eye-tracking sensor can be selected from 60 to 240 times per second, the key point extraction frequency of the gesture sensor can be selected from 30 to 100 times per second, and the environmental sensor can be configured in the range of 1 to 10 times per second to reduce unnecessary communication load. In implementation, multiple data streams can include rendered video frame streams, spatial audio streams, and multiple packaged sensor streams. Each type of stream is assigned an independent identifier and carries a transmission priority weight in the transmission stack, which facilitates differentiation during multi-path scheduling and synchronous processing. Optionally, for resource-constrained terminals, the sampling frequency of gesture data and environmental data can be reduced while maintaining high sampling of inertial and eye-tracking data to ensure that the overall computing power and link bandwidth can meet the minimum real-time requirements. When a certain type of sensor experiences a temporary failure or data loss, the system can use the most recent valid measurement value or a short-term extrapolation result based on inertial data to replace it within several sampling periods, ensuring that the entire processing flow from S1 to S5 can still be executed continuously without causing the synchronization module to fail.

[0053] In one embodiment, fusing multi-sensor data includes:

[0054] Based on the extended Kalman filter algorithm, the state prediction and update of multi-sensor data output by eye-tracking sensor, inertial measurement unit sensor, gesture sensor and environmental sensor are performed to generate a unified user behavior vector. The user behavior data includes the user behavior vector.

[0055] When fusing multi-sensor data output from eye-tracking sensors, inertial measurement unit sensors, gesture sensors, and environmental sensors:

[0056] The system aligns multi-sensor data along the time axis and constructs a user behavior state vector that simultaneously represents the user's head spatial position, head linear velocity, head posture, gaze direction, and gesture. In the state prediction stage, the user behavior state vector from the previous moment and the output of the inertial measurement unit (IMU) sensor are used as inputs. A nonlinear state transition model based on head kinematics, gaze dynamics, and gesture dynamics is used for state extrapolation, and the prediction uncertainty is modeled by incorporating process noise covariance. In the state update stage, a joint observation model is constructed based on the measurement mechanisms of eye-tracking sensors, IMU sensors, gesture sensors, and environmental sensors. The predicted state is mapped to the observation space of each sensor. The predicted state is corrected through innovative extended Kalman filtering, Kalman gain, and covariance update steps, resulting in a unified user behavior vector updated by multi-sensor fusion, which is output as part of the user behavior data.

[0057] Specifically, a unified user behavior vector can be implemented by maintaining a fixed-length real-number array in memory. Each element corresponds to a dimension of head position, linear velocity, pose, gaze direction, and gesture encoding, with the array length consistent with the state dimensions. In engineering implementation, this user behavior vector can be updated frame-by-frame over time and passed to subsequent priority evaluation and transmission scheduling modules via a shared memory queue or lightweight message queue, avoiding repeated decoding from the original sensor data. The vector update interval can be consistent with the sampling period of the multi-sensor system, typically ranging from 5 milliseconds to 20 milliseconds, thus ensuring smooth behavior estimation while considering the overall computational load. In terms of memory layout, it can optionally be stored in segmented order according to position, velocity, pose, gaze, and gesture, facilitating quick indexing of required fields by different modules. For simplified scenarios that do not require all dimensions, some dimensions can be fixed to zero or retain the values ​​from the previous moment to reduce computational overhead without changing the unified vector interface.

[0058] The following is a method for state prediction and updating based on the extended Kalman filter algorithm for multi-sensor data:

[0059] S21, After completing the time alignment and coordinate unification of the raw data from each sensor, the user behavior state vector is constructed in step S2, as follows:

[0060] ,

[0061] in, Representing discrete time The user behavior state vector Indicates time The position vector of the user's head in three-dimensional space, in meters (m). Indicates time The linear velocity vector of the user's head, in m / s. Indicates time The user's head pose vector can be expressed as Euler angles or equivalent pose parameters, in rad. Indicates time The user's gaze direction-related vector can be a unit direction vector or the coordinates of the gaze point on the display plane. Indicates time Low-dimensional encoded vectors of user gestures, such as joint angle features or embedded features extracted from gesture sensors, superscript This indicates that the vector is transposed. Represents a discrete-time index, which is a non-negative integer;

[0062] For example, the user's head position vector can be represented in a 3D world coordinate system, where the origin can be placed at a reference point in the virtual scene or physical space, with the unit of position being meters. The head linear velocity can be estimated by the temporal difference of the position vector combined with the acceleration output of the inertial measurement unit, with the unit being meters per second. Head pose can be encoded using Euler angles or equivalent quaternions; in software implementation, one of these methods can be chosen based on the existing pose representation methods of the rendering engine to avoid unnecessary coordinate transformations. The gaze direction-related vector can be obtained by geometrically transforming the pupil position output by the eye-tracking sensor combined with the current head pose, or by directly using the unit direction vector provided by the device driver. The gesture action encoding vector can be obtained by mapping high-dimensional hand keypoint features to low-dimensional embeddings during the preprocessing stage of the gesture recognition module. The embedding dimension can be selected from 16 to 64 dimensions to balance expressive power and computational complexity. In actual deployment, the discrete-time index can correspond one-to-one with the sampling period. For example, when the sampling period is ten milliseconds, the index advances by ten milliseconds for each corresponding time increment, facilitating time tracking during logging and debugging.

[0063] S22, in step S2, the various sensor measurements collected and time-aligned in step S1 are concatenated into a unified observation vector:

[0064] ,

[0065] in, Indicates time Multi-sensor observation vectors, This indicates that the eye-tracking sensor is at time [time]. The output observation vectors, such as pupil center position and eye rotation angle, are features. Indicates the inertial measurement unit sensor at time [time]. The output observation vectors, such as triaxial acceleration and triaxial angular velocity, Indicates the gesture sensor at any time The output observation vectors, such as hand keypoints or multi-joint angle features, Indicates the environmental sensor at time The output observation vectors, such as temperature and light intensity, are superscripted. Indicates the transpose operation, superscript , , , Used to distinguish different sensor sources;

[0066] Similarly, multi-sensor observation vectors can be constructed by sequentially concatenating the raw or pre-processed outputs of different types of sensors, maintaining a fixed order for index-based parsing in the observation function. Eye-tracking observation vectors can include fields such as pupil center coordinates, pupil size, and eye rotation angle, which can be selected based on the interface provided by the equipment manufacturer. Inertial measurement observation vectors can include triaxial acceleration and triaxial angular velocity, with values ​​generally within the upper and lower limits of gravitational acceleration and angular velocity specified by the equipment specifications. Gesture observation vectors can be represented by the three-dimensional coordinates of several key hand points or skeletal joint angles. Environmental observation vectors can include scalar information such as calibrated temperature and light intensity. Each observation component can be normalized or standardized during acquisition, for example, by subtracting the mean and dividing by the standard deviation, compressing the numerical range to a range conducive to stable numerical calculation. Optionally, when a certain type of sensor has no new measurement value at a certain time point, the observation value from the previous moment can be copied and filled, or a placeholder value can be added to the observation vector, and a larger uncertainty can be assigned to that component when modeling observation noise.

[0067] S23, In the state prediction phase, the posterior state of the previous time step and the IMU measurement are used as control inputs to extrapolate the state at the current time step:

[0068] ,

[0069] in, Indicates at time The prior state vector, given that it is used at time [time]. Predictions are made based on observations. Indicates time The posterior state vector, This represents a nonlinear state transition function based on user head kinematics, gaze dynamics, and gesture dynamics, used to give the state transition from time [time]. At the time State prediction, Indicates the inertial measurement unit at time... The output is a control input vector, such as a combination of acceleration and angular velocity. Indicates from time At the time The time interval, in seconds. Indicates time The state-process noise vector is used to model the unmodeled dynamics of the system and external disturbances. The subscript... Indicates at time The estimation results are conditional upon time. Observations, subscript Indicates at time The estimation results are conditional upon time. Observations;

[0070] For state-process noise, a Gaussian distribution can be used for modeling:

[0071] ,

[0072] in, Indicates time State process noise vector, Indicates a multivariate Gaussian distribution. Represents a vector with zero mean. This indicates that time is 1 in the extended Kalman filter process. The process noise covariance matrix can be set based on historical statistics or online estimation. (Superscript) This indicates that the covariance matrix belongs to the extended Kalman filtering process;

[0073] In this embodiment, the process noise covariance matrix Q can be initialized based on historical motion data and device specifications. For example, the diagonal elements related to head position and velocity can be set to the order of square centimeters to square decimeters, the diagonal elements related to posture and line of sight can be set to suitable small values ​​in angular units, and the gesture encoding related elements can be set according to the output fluctuation range of the gesture recognition module. The observation noise covariance matrix R can be estimated based on the measurement accuracy specifications provided by each sensor manufacturer or the online calibration process. In engineering, the noise levels of eye-tracking and inertial measurements are usually set to be low, while the noise levels of gesture and environmental measurements are set to be relatively high to reflect the reliability of different observation sources. The initial value of the state covariance matrix P can be a diagonal matrix, with the diagonal elements providing the upper limit of uncertainty for initial position, velocity, posture, line of sight, and gesture. It then automatically converges to a smaller value during the filtering process. Optionally, when the system detects a long-term sensor anomaly or a restart event, the covariance matrix can be reset to a larger diagonal value to avoid the continuous impact of historical erroneous estimates on subsequent behavior vectors.

[0074] The prior prediction of the state covariance can be written as:

[0075] ,

[0076] in, Indicates time The prior state covariance matrix, Indicates the state For nonlinear state transition functions The Jacobian matrix obtained by linearizing the state vector. Indicates time The posterior state covariance matrix, Representing the Jacobian matrix The transpose of the matrix, Indicates time The process noise covariance matrix is ​​consistent with the aforementioned process noise distribution, and the superscript... This matrix indicates that it is used for extended Kalman filtering;

[0077] S24, In the observation modeling phase, the state vector is mapped to multi-sensor observations, written as:

[0078] ,

[0079] in, Indicates time The multi-sensor observation vector is consistent with the observation vector in step S22. This represents the global nonlinear observation function from the state space to the observation space, used to give the state... The following prediction observations Indicates time The prior state vector, Indicates time The observed noise vector, subscript Indicates prior estimate, superscript Used to distinguish observation noise from other types of noise;

[0080] Depending on the sensor type, the observation function can be decomposed into:

[0081] ,

[0082] in, This indicates that the global observation function is in the prior state. The output at that location, This represents a sub-function that derives the observables of the eye-tracking sensor based on the prior state. This represents a sub-function that derives the IMU observables based on attitude and velocity components. This represents a sub-function that derives the observable quantities of the gesture sensor based on the gesture components. This represents a subfunction that derives the observable quantities of an environmental sensor based on its position and other components; the superscript indicates that the function is not explicitly defined. , , , Used to distinguish between different sub-observation functions Indicates a discrete-time index;

[0083] Observation noise can also be modeled using a Gaussian distribution:

[0084] ,

[0085] in, Indicates time The observed noise vector, Indicates a multivariate Gaussian distribution. Represents the zero vector. This indicates that time is 1 in the extended Kalman filter process. The observed noise covariance matrix can be set according to the noise statistical characteristics of each sensor, with superscript... This indicates that the matrix belongs to the extended Kalman filter process;

[0086] In order to use a linear update model in the extended Kalman filter, the observation function needs to be linearized near the prior state:

[0087] ,

[0088] in, Indicates the prior state observation function The Jacobian matrix obtained by linearizing the state vector. Indicates time The prior state vector, This represents the predicted observation vector under the prior state, with the symbol... Indicates approximate equality in the sense of a first-order Taylor expansion, with superscript. This indicates that the Jacobian matrix is ​​used for extended Kalman filtering;

[0089] S25, After obtaining actual and predicted observations, calculate the innovation vector:

[0090] ,

[0091] in, Indicates time The innovation vector, which is the difference between actual observations and predicted observations, Indicates time The actual multi-sensor observation vector, Represents the predicted observation vector calculated under the prior state, with the subscript... The wavy line symbol represents a priori estimation. This vector represents the innovation quantity;

[0092] The innovation covariance matrix can be written as:

[0093] ,

[0094] in, Indicates time The innovation covariance matrix is ​​used to characterize the uncertainty of the innovation vector. Indicates time The observed Jacobian matrix, Indicates time The prior state covariance matrix, This represents the transpose of the observation Jacobian matrix. Represents the observation noise covariance matrix, with superscript... This indicates that the above matrices all belong to the extended Kalman filtering process;

[0095] The Kalman gain matrix is ​​calculated as follows:

[0096] ,

[0097] in, Indicates time The extended Kalman gain matrix is ​​used to balance the weights of prior predictions and observational information. Represents the prior state covariance matrix. This represents the transpose of the observation Jacobian matrix. The matrix representing the inverse of the innovation covariance matrix is ​​indicated by the superscript. This indicates that the gain matrix belongs to the extended Kalman filter process. Indicates a discrete-time index;

[0098] Update the prior state using Kalman gain:

[0099] ,

[0100] in, Indicates time The posterior state vector is the unified user behavior estimation result after fusing observations from multiple sensors. Indicates time The prior state vector, Indicates time The extended Kalman gain matrix, Indicates time Innovation vector, subscript This indicates that the state estimation condition is at time [time]. All observations;

[0101] The posterior update of the state covariance can be written as:

[0102] ,

[0103] in, Indicates time The posterior state covariance matrix, The dimension is The identity matrix is ​​used to maintain consistent matrix dimensions. State vector Dimensions Indicates time The extended Kalman gain matrix, Indicates time The observed Jacobian matrix, Indicates time The prior state covariance matrix, superscript This indicates that these matrices belong to the extended Kalman filter process;

[0104] S26, After completing the state update, this embodiment outputs the posterior state vector as a unified user behavior vector to generate a unified user behavior vector:

[0105] ,

[0106] in, Indicates time The output user behavior vector is used to simultaneously represent the user's posture, gaze direction, and gesture actions. Indicates time The posterior state vector, index Indicates a discrete-time index;

[0107] Specifically, this implementation scheme employs a layered design for the multi-sensor data fusion process, focusing on the extended Kalman filtering step. First, through state vectors, the user's head position, linear velocity, posture, gaze direction, and gestures are encoded into a unified behavior space. This ensures that subsequent processing no longer directly deals with heterogeneous sensor formats but utilizes a clearly structured behavior state. Then, through observation vectors, the outputs from eye tracking, inertial measurement, gesture, and environmental sensors are uniformly organized and a nonlinear mapping relationship is established with the state space. Based on this, the calculation of predictive observations is completed. In the state prediction stage, the fusion results from the previous moment are extrapolated using the inertial measurement input, and a process noise covariance constraint model is applied. Errors are mitigated to keep the state smooth over time without excessive divergence. During the observation phase, the observation Jacobian matrix is ​​obtained through linearization. Then, innovation, innovation covariance, and Kalman gain are used to weight and combine the prior state with multi-source observations, giving sensors with lower noise and higher reliability a greater weight in the fusion result. The updated posterior state covariance reflects the uncertainty level of the current estimate, providing a statistical basis for subsequent dynamic adjustment of transmission priorities. The final output user behavior vector can be directly used as a unified input for subsequent priority evaluation, data stream scheduling, and synchronization processing, which helps reduce the temporal and spatial deviations of different modal data and reduce user-perceived posture jitter and line-of-sight misalignment.

[0108] Optionally, in the specific software implementation, the prediction and update frequency of the extended Kalman filter can be aligned with the system rendering frame rate or audio / video frame rate. For example, a state prediction can be triggered at the beginning of each frame rendering, and a state update can be completed after all available observations for the current frame are obtained. The filtering operation can be performed in an independent thread on the edge computing node or terminal device, and data interaction with the rendering thread can be achieved through a lock-free queue or circular buffer to avoid blocking the rendering pipeline. For terminals with limited computing resources, the state update frequency can be appropriately reduced, such as from 100 times per second to 50 times per second, and the trajectory smoothness can be maintained by adjusting the process noise covariance. If filter divergence or increased jitter is found during operation, diagnostic information can be reported to the edge node to request recalibration of noise parameters or reset of the state, thereby maintaining feasibility without changing the overall algorithm structure.

[0109] In one embodiment, determining the transmission priority of various data streams based on user behavior data includes:

[0110] Using user behavior data as input sequence, the system uses a long short-term memory neural network model to predict user interaction hotspots in a virtual scene, and calculates transmission priority weights for different sensor data streams based on these hotspots.

[0111] Inputting user behavior data into a long short-term memory neural network model includes:

[0112] User behavior vectors within a predetermined time window are collected on the time axis and concatenated with contextual features obtained from environmental sensors and virtual scene grid location encoding to form the input sequence of a Long Short-Term Memory (LSTM) neural network. This LSM network is then used to perform temporal modeling of the input sequence, yielding the probability distribution of interaction hotspots across multiple spatial regions defined in the virtual scene. For each type of sensor data stream, correlation parameters between the sensor and each spatial region are pre-defined. The unnormalized priority score of the sensor data stream is calculated based on the interaction hotspot probability distribution and the corresponding correlation parameters. Through interval compression and normalization, the unnormalized priority scores of various sensor data streams are converted into comparable transmission priority weights at the same scale. During model training, a joint loss function is used, including a cross-entropy term based on interaction region annotations, a priori distribution constraint term based on user gaze generation, and a time smoothing term that penalizes changes in the probability distribution of hotspots at consecutive moments, to jointly optimize the LSM network model and correlation parameters, thereby improving the stability and robustness of interaction hotspot prediction and transmission priority allocation.

[0113] The steps for predicting interactive hotspots and mapping transmission priority weights using LSTM include:

[0114] S31, to characterize recent user behavior and environmental context, constructs time-space parameters. The fused input vectors are formed into a length of sequence:

[0115] ,

[0116] in, Indicates time The LSTM single-step input vector, This represents the user behavior vector from step S2. This represents a context feature vector, containing an environmental sensor summary and scene grid location encoding; the values ​​have been normalized. This indicates a dimensional concatenation operation. Indicates length is The input sequence, The time window length (number of frames) is represented by a positive integer, and the index is... Indicates the index of discrete time points;

[0117] S32, using single-layer or multi-layer LSTM to... Modeling is performed (the following is a single-layer recursive formula):

[0118] ,

[0119] in, Represents the input gate vector. This represents the Sigmoid function. This represents the weight matrix input to the input gate. This represents the weight matrix from the hidden state to the input gate. This represents the input gate bias vector. Indicates time The input vector, This represents the hidden state vector from the previous step.

[0120] ,

[0121] in, Represents the forget gate vector. This represents the weight matrix input to the forget gate. This represents the weight matrix from the hidden state to the forget gate. Represents the forget gate bias vector;

[0122] ,

[0123] in, Represents the candidate cell state vector. Represents the hyperbolic tangent function. This represents the weight matrix input to the candidate state. This represents the weight matrix from the hidden state to the candidate state. Represents the candidate state bias vector;

[0124] ,

[0125] in, Represents the cell state vector. This indicates Hadamard's element-wise multiplication. This represents the cell state vector from the previous step. , Same definition as above;

[0126] ,

[0127] in, Represents the output gate vector. , , These represent the weight matrix and bias corresponding to the output gate, respectively. This represents the hidden state vector, which is the input for subsequent decoding;

[0128] S33, mapping the hidden states to probability distributions on the scene mesh (total) (regions)

[0129] ,

[0130] in, Indicates time The probability distribution vector of hotspot regions, Represents the normalized exponent mapping, The weight matrix representing the distribution from hidden states to hotspots. This represents the bias vector. Indicates the number of regions. Indicates the first The probability of each region;

[0131] S34, for each type of data stream (like , , , Calculate the unnormalized weights and normalize them to obtain the priority:

[0132] ,

[0133] in, Indicates time Unnormalized priority score, Indicates sensor Compression mapping function, Indicates sensor offset item, Indicates the area For sensors The correlation coefficient, Indicates the area The probability, Indicates the number of regions. Represents a set of sensors;

[0134] ,

[0135] in, Indicates the real number Mapping to interval The function, and These represent the sensors. The lower and upper limits of the score, Represents the Sigmoid function;

[0136] ,

[0137] in, Indicates time Assigned to sensor Normalized transmission priority weights Represents unnormalized fractions. This represents the summation of scores from all sensors. Represents a set of sensors;

[0138] S35 is jointly trained using label-smoothed cross-entropy, along with the gaze prior KL term and temporal smoothing term:

[0139] ,

[0140] in, Represents cross-entropy loss, This represents the supervised probability after label smoothing. Represents the smoothing coefficient, with a range of values. , Indicates the area A hot or soft label, This represents the model output probability. Indicates the number of regions. Indicates a time index;

[0141] ,

[0142] in, The KL divergence term is represented by the gaze prior distribution. This indicates the line of sight in step S2. Prior probabilities of the generated region Indicates the model output probability;

[0143] ,

[0144] in, Indicates the first The coordinate vectors of the center of each region on the display plane, in pixels or normalized coordinates. This indicates the direction vector of the line of sight. The projection function projected onto the display plane. Represents the L2 norm, This represents the Gaussian kernel width parameter, with units consistent with the coordinate system. For summation index;

[0145] ,

[0146] in, Represents the time smoothing term. and These represent the hotspot distribution at adjacent time points. Represents the L2 norm;

[0147] ,

[0148] in, Indicates the total loss. , , These represent the weighting coefficients for the three losses, and their values ​​are non-negative real numbers.

[0149] Scene mesh is desirable or , It can include segmented encoding of ambient brightness and discretized temperature indication to avoid dimensional interference. Use 8–20 frames to balance latency and stability. and It is recommended to set it to interval, It can be obtained offline from task statistics or online lightweight adaptive regression. This will be directly referenced by multipath scheduling in step S4;

[0150] Specifically, in step S3, user interaction intent is extracted using time-series modeling. The input consists of the behavior vector output from step S2 and the environmental context, forming a fixed window sequence for LSTM processing. Gated recursion captures short-term and medium-term dependencies, thus providing stable hotspot predictions during user gaze shifts and gesture preparation. Hotspots are output in a gridded probability format, facilitating integration with subsequent encoding, rendering, and transmission modules. Priority mapping is not directly bound to a specific sensor but rather uses regional correlation coefficients to linearly aggregate the values ​​of various sensors, followed by interval compression and normalization to obtain weights, facilitating comparable allocation under total bandwidth constraints. Label smoothing is introduced during training to reduce hard label noise, and the KL term between the gaze-generated prior and the model output is used to enhance interpretability. Temporal smoothing is used to suppress rapid jitter and avoid frequent path switching at trajectory boundaries. This combined loss strikes a balance between stability and responsiveness, providing continuous and differentiable weight inputs for multi-path scheduling in the subsequent step S4, thereby maintaining data availability and screen continuity in interactive hotspot areas under link fluctuation conditions.

[0151] Furthermore, the dataset used to train the Long Short-Term Memory (LSTM) network and its relevance parameters can be obtained by collecting interaction trajectories of real users in virtual collaborative scenarios. Training data can include records of user head movements, gaze changes, and gestures under different tasks, as well as interactive hotspot labels generated manually or by semi-automatic tools. The number of training samples can be set according to model complexity, typically ranging from tens of thousands to hundreds of thousands of time series segments. The length of each time series segment should be consistent with or an integer multiple of the time window length. To improve the model's generalization ability to different users and different lighting environments, the training data can include samples under various ambient brightness and temperature conditions, and these samples can be preprocessed. Contextual features are normalized. Model training can be completed on offline servers or edge computing clusters. The obtained network parameters are deployed to edge nodes or terminal devices for forward inference at runtime, without performing backpropagation and parameter updates during runtime, thus ensuring controllable runtime computation. If computing power is limited in some deployment environments, the hidden layer dimension can be reduced, the time window length can be shortened, or a single-layer long short-term memory network can be used, while maintaining the input / output interfaces and the meaning of priority weights unchanged.

[0152] In one embodiment, the environmental sensor includes a temperature sensor and a light sensor, and the method further includes:

[0153] Based on the environmental information collected by temperature and light sensors, the transmit power, coding rate, and retransmission strategy in multipath transmission scheduling are adjusted to obtain a data transmission strategy that meets the current environmental conditions.

[0154] In this embodiment, ambient temperature and illumination information can be periodically reported by temperature probes and illumination sensors built into or connected to the VR panoramic glasses. The sampling period can be set between 1 and 10 seconds to avoid additional burden on the link. Based on the temperature readings, the system can appropriately reduce the upper limit of transmission power or the encoding bitrate in high-temperature scenarios to control the power consumption and heat generation levels of the terminal and edge nodes; when the temperature is low and the link quality is good, a higher encoding bitrate can be allowed to improve image quality. Illumination intensity information can be used to infer the user's visual perception sensitivity to details. When the illumination is weak, the encoding strategy can be biased towards preserving key contours and motion information, while when the illumination is strong and the screen brightness is high, the encoder can be allowed to sacrifice some low-value texture details in exchange for lower latency. Optionally, when the environmental sensor experiences a temporary anomaly or the reading is unreliable, the system can fall back to a set of default transmission strategies and re-enable the environmental adaptive adjustment logic after several sampling periods.

[0155] In one embodiment, multipath transport scheduling includes:

[0156] The first transmission path carrying high-priority data streams is configured as an ultra-reliable low-latency communication channel, and the second transmission path carrying low-priority data streams is configured as a high-capacity wireless channel.

[0157] For example, the first transmission path can be mapped in actual deployment to a service bearer channel configured for low-latency scenarios in the operator's network, or a high-priority quality of service queue in the local area network. The second transmission path can be mapped to a regular data channel with sufficient bandwidth but allowing for higher queuing latency. Path selection can be determined by signaling negotiation during the connection establishment phase, or it can be dynamically switched during operation based on link quality feedback. However, in this embodiment, the mapping relationship between high-priority and low-priority data streams remains unchanged to avoid additional jitter caused by frequent switching. To simplify implementation, two logical paths can be simulated on the same physical link using different quality of service tags or priority queues. The high-priority queue is configured with a smaller queuing depth and a tighter scheduling cycle, while the low-priority queue is configured with a larger buffer space to absorb bandwidth fluctuations. Optionally, when severe congestion is detected in the second transmission path and it affects the overall experience, the transmission frequency or resolution of low-priority data can be temporarily reduced without affecting the transmission configuration of high-priority data streams in the first transmission path.

[0158] In one embodiment, synchronizing multiple data streams obtained through multipath transmission includes:

[0159] Assign logical clock timestamps to data streams from multiple VR panoramic glasses;

[0160] Sort and buffer multiple data streams based on logical clock timestamps;

[0161] The playback timing of each data stream is adjusted based on the sorting results to achieve synchronization of multi-user data streams;

[0162] Specifically, the logical clock timestamp can be implemented by assigning a uniform initial count value to each VR panoramic glasses at system startup, and incrementing it by a fixed increment at each discrete time step. This discrete time step can be consistent with the behavior vector update cycle, for example, once every ten milliseconds. After receiving data streams from different users, the synchronization processing module can first perform coarse sorting based on the logical clock timestamps, and then determine the final playback order and latency compensation amount by combining the actual arrival time and the maximum allowed buffer duration. The maximum buffer duration can be set according to the target interaction latency and the user's acceptable level of stuttering, with a typical value between fifty milliseconds and two hundred milliseconds, to achieve a balance between synchronization effect and real-time performance. Optionally, when a user's data stream has not arrived within a preset buffer time window, the system can choose to use the user's behavior vector or image from the previous moment for frame interpolation and discard expired frames when subsequent data arrives, to avoid the overall collaborative scene from being interrupted due to individual link anomalies.

[0163] In one embodiment, the method is executed at least in part on an edge computing node, which is used to perform fusion processing of multi-sensor data and determine transmission priorities;

[0164] In one embodiment, the method further includes:

[0165] The transmitted data will be fragmented into multiple data blocks and sent.

[0166] When network packet loss is detected, the incomplete data blocks from multiple data blocks are input into a pre-trained generative adversarial network model. The generative adversarial network model reconstructs the lost data blocks and prioritizes the recovery of data blocks carrying high-priority data streams during reconstruction.

[0167] Furthermore, in the implementation, data blocks can be divided into encoded video segments or key sensor data according to a fixed size. For example, each block corresponds to several rows of pixels or a coded segment of a certain time length. The block size can be configured in the range of several thousand bytes to tens of thousands of bytes to finely control the recovery granularity during packet loss detection and retransmission. The input to the generative adversarial network model can be the available part of the received incomplete block plus the context information of adjacent temporal or spatial locations. The output is a reconstruction result with the same size as the original block. The training process can be completed offline using a large number of samples with artificially introduced packet loss, and the difference between the samples and the original data without packet loss can be used as a supervision signal. In actual operation, when the reconstruction time exceeds the preset limit or the device's computing power is insufficient to complete the full GAN ​​forward inference within the current frame time budget, a simpler interpolation or copying strategy can be used to temporarily replace the reconstruction result to ensure that the playback sequence is not disrupted. At the same time, this situation is recorded as diagnostic information for subsequent optimization.

[0168] In one embodiment, multipath transmission scheduling of at least two physical or logical transmission paths based on transmission priority includes:

[0169] Different lengths of send buffer queues are allocated to data streams with different transmission priorities. The length of the send buffer queue corresponding to a high-priority data stream is less than the length of the send buffer queue corresponding to a low-priority data stream.

[0170] In this embodiment, the length of the sending buffer queue can be measured by the number of data packets to be sent or the cumulative data volume. The specific value can be determined based on the link bandwidth and target latency. For example, the high-priority queue can be limited to a few dozen data packets, while the low-priority queue can allow hundreds of data packets to queue. The queue length can be adaptively adjusted during system operation based on the monitored link latency and packet loss rate. When the queuing time of high-priority data packets is detected to be close to a preset threshold, queue space can be released by reducing the injection rate of low-priority data or actively discarding some expired low-priority data. Optionally, in cases of extremely limited resources or severe link congestion, the system can temporarily suspend the sending of low-priority queues, retaining only the transmission of high-priority data streams to ensure the timely delivery of critical interactive information. When the link condition recovers, the sending rate of low-priority queues is gradually restored to avoid sudden traffic spikes.

[0171] In one embodiment, the method further includes:

[0172] Based on the link latency, packet loss rate and available bandwidth measured during multipath transmission, the transmission priority weight or multipath transmission scheduling strategy is periodically updated to reduce end-to-end latency fluctuations of high-priority data streams.

[0173] Optionally, link latency, packet loss rate, and available bandwidth can be obtained by periodically sending probe messages or statistically analyzing the transmission results of actual business data. The update period can be set between tens and hundreds of milliseconds to balance the timeliness of feedback and measurement overhead. Priority weight updates can employ a smoothing filter, weighting the new estimate with the previous weight to avoid frequent oscillations in the transmission strategy due to instantaneous measurement anomalies. Adjustments to the multi-path scheduling strategy can include temporarily increasing the weight of certain sensor flows under conditions of high packet loss or high latency, or reallocating some high-priority flows to high-capacity paths to reduce interference with high-priority flows. In extreme cases, if a path is detected to be unavailable for an extended period, all data flows can be migrated to the remaining available paths, and different priority data flows can continue to be distinguished internally using quality of service (QoS) marking to ensure the overall system still has degraded operation capabilities.

[0174] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0175] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of this application and form different embodiments. For example, all the embodiments above can be used in any combination. The information disclosed in this background section is intended only to enhance the understanding of the general background of this application and should not be construed as an admission or in any way implying that such information constitutes prior art known to those skilled in the art.

Claims

1. A method for data synchronization and transmission of VR panoramic glasses based on multi-sensor fusion, applied to a virtual collaborative system including VR panoramic glasses and edge computing nodes, wherein the VR panoramic glasses include multiple sensors, characterized in that, The method includes: S1, Collect multi-sensor data output by the multiple sensors, wherein the multiple sensors include at least an eye-tracking sensor, an inertial measurement unit sensor, a gesture sensor, and an environmental sensor; S2, the multi-sensor data is fused to generate user behavior data representing user posture, gaze direction and gestures; S3, determine the transmission priority of various data streams based on the user behavior data; S4, perform multi-path transmission scheduling on at least two physical or logical transmission paths according to the transmission priority, and allocate high-priority data streams and low-priority data streams to different transmission paths respectively. S5, perform synchronization processing on the multiple data streams obtained through the multi-path transmission to align the playback timing of the multiple data streams; The fusion of the multi-sensor data includes: Based on the extended Kalman filter algorithm, the state prediction and update of the multi-sensor data output by the eye-tracking sensor, inertial measurement unit sensor, gesture sensor and environmental sensor are performed to generate a unified user behavior vector, and the user behavior data includes the user behavior vector; When fusing multi-sensor data output from the eye-tracking sensor, inertial measurement unit sensor, gesture sensor, and environmental sensor: The system aligns multi-sensor data along the time axis and constructs a user behavior vector that simultaneously represents the user's head spatial position, head linear velocity, head posture, gaze direction, and gesture. In the state prediction stage, the user behavior vector from the previous moment and the output of the inertial measurement unit (IMU) sensor are used as inputs. A nonlinear state transition model based on head kinematics, gaze dynamics, and gesture dynamics is used for state extrapolation, and the prediction uncertainty is modeled by incorporating process noise covariance. In the state update stage, a joint observation model is constructed based on the measurement mechanisms of eye-tracking sensors, IMU sensors, gesture sensors, and environmental sensors. The predicted state is mapped to the observation space of each sensor. The predicted state is corrected through innovative extended Kalman filtering, Kalman gain, and covariance update steps, resulting in a unified user behavior vector updated by multi-sensor fusion, which is output as part of the user behavior data. Determining the transmission priority of various data streams based on the user behavior data includes: The user behavior data is used as an input sequence. A long short-term memory neural network model is used to predict the user's interaction hotspots in the virtual scene, and the transmission priority weights for different sensor data streams are calculated based on the interaction hotspots. Inputting the user behavior data into a long short-term memory neural network model includes: User behavior vectors within a predetermined time window are collected on the time axis and concatenated with contextual features obtained from environmental sensors and virtual scene grid position encoding to form the input sequence of a Long Short-Term Memory (LSTM) neural network. The LSM neural network is then used to perform temporal modeling on the input sequence to obtain the probability distribution of interaction hotspots in multiple spatial regions divided within the virtual scene. For each type of sensor data stream, a correlation parameter between the sensor and each spatial region is pre-defined. Based on the interaction hotspot probability distribution and the corresponding correlation parameter, the unnormalized priority score of the sensor data stream is calculated. Through interval compression and normalization, the unnormalized priority scores of various sensor data streams are converted into comparable transmission priority weights at the same scale. During model training, a joint loss function is used, including a cross-entropy term based on interaction region annotation, a priori distribution constraint term based on user gaze generation, and a time smoothing term that penalizes changes in the hotspot probability distribution at consecutive moments, to jointly optimize the LSM neural network model and the correlation parameters. The multipath transmission scheduling includes: The first transmission path carrying high-priority data streams is configured as an ultra-reliable low-latency communication channel, and the second transmission path carrying low-priority data streams is configured as a high-capacity wireless channel. Synchronization processing of the multiple data streams obtained through the multipath transmission includes: Assign logical clock timestamps to data streams from multiple VR panoramic glasses; The multiple data streams are sorted and buffered according to the logical clock timestamp; The playback timing of each data stream is adjusted based on the sorting results to achieve synchronization of multi-user data streams; The method is executed at least in part on the edge computing node, which is used to perform the fusion processing of the multi-sensor data and the determination of the transmission priority.

2. The VR panoramic glasses data synchronization and transmission method based on multi-sensor fusion as described in claim 1, characterized in that, The environmental sensor includes a temperature sensor and a light sensor, and the method further includes: Based on the environmental information collected by the temperature sensor and the light sensor, the transmit power, coding rate and retransmission strategy in the multipath transmission scheduling are adjusted to obtain a data transmission strategy that meets the current environmental conditions.

3. The VR panoramic glasses data synchronization and transmission method based on multi-sensor fusion as described in claim 2, characterized in that, The method further includes: The data will be fragmented into multiple data blocks and sent. When network packet loss is detected, the incomplete data blocks from the plurality of data blocks are input into a pre-trained generative adversarial network model, which reconstructs the lost data blocks and prioritizes the recovery of data blocks carrying high-priority data streams during reconstruction.

4. The VR panoramic glasses data synchronization and transmission method based on multi-sensor fusion as described in claim 3, characterized in that, Multipath transmission scheduling of at least two physical or logical transmission paths based on the transmission priority includes: Different lengths of send buffer queues are allocated to data streams with different transmission priorities. The length of the send buffer queue corresponding to a high-priority data stream is less than the length of the send buffer queue corresponding to a low-priority data stream.

5. The VR panoramic glasses data synchronization and transmission method based on multi-sensor fusion as described in claim 4, characterized in that, The method further includes: Based on the link delay, packet loss rate, and available bandwidth measured during the multipath transmission process, the transmission priority weight or the multipath transmission scheduling strategy is periodically updated.

Images