A multi-vr terminal cooperative interaction system and method based on mutual positioning calculation

Abstract

This application provides a multi-VR terminal collaborative interaction system and method based on mutual positioning computation, relating to the field of VR terminal collaborative interaction technology. It involves constructing a collaborative perception network; evaluating the pose reliability of the virtual character mapped to each VR terminal in the collaborative perception network to obtain the observation reliability of each virtual character's pose; when all VR terminals detect a virtual character with lost visual observation, the collaborative perception network broadcasts the last observation information of the lost virtual character to determine the pose constraint boundary of the lost virtual character during the VR terminal occlusion interruption; predicting the trajectory of the skeletal keypoints of the lost virtual character to obtain the predicted motion trajectory of the virtual character's skeletal keypoints; and performing a pose transition on the lost virtual character to obtain the virtual character's pose transition parameters, thereby adjusting the virtual character's spatial position. This application can prevent visual jumps in virtual characters when VR terminal occlusion interruption occurs.

Description

Technical Field

[0001] This application relates to the field of VR terminal collaborative interaction technology, and more specifically, to a multi-VR terminal collaborative interaction system and method based on mutual positioning calculation. Background Technology

[0002] With the rapid development of Virtual Reality (VR) technology, multi-user immersive collaborative interaction has become a key direction for VR applications. VR terminal collaborative interaction technology greatly enhances the immersion and realism of collaborative training, social entertainment, and remote conferencing by enabling real-time interaction among multiple users in the same virtual space. The core of this technology lies in using the mutual positioning between terminals, such as data exchange based on vision and ranging, to build a distributed perception network, thereby accurately synchronizing the position and posture of each user's virtual character and ensuring the consistency of the virtual world state from the perspective of all participants. This provides a crucial technical foundation for building a shared and coherent virtual experience.

[0003] In existing technologies, multiple VR terminals typically rely on continuous mutual visual observation to maintain the stability and pose synchronization of the collaborative perception network. However, in dynamic and complex interactive environments, physical occlusion, rapid movement, or limited field of view between users can easily cause one terminal to temporarily lose visual tracking of the corresponding virtual character on other terminals. Once such observation interruption occurs, existing methods often lack effective pose maintenance and transition mechanisms. They either rigidly maintain the character at the last observed position, creating a "frozen" illusion, or use simple linear interpolation to predict its movement during occlusion. Both of these approaches deviate from the actual movement trajectory of the virtual character. When vision resumes, the position and posture of the virtual character will abruptly change, producing obvious visual jumps and a sense of spatial dislocation, which seriously undermines the immersive experience and spatial consistency of multi-user collaboration. Therefore, how to avoid visual jumps in virtual characters when occlusion interruption occurs in VR terminals has become a difficult problem for the industry. Summary of the Invention

[0004] This application provides a multi-VR terminal collaborative interaction system and method based on mutual positioning calculation, which can prevent virtual characters from visually jumping when VR terminal is occluded and interrupted.

[0005] In a first aspect, this application provides a multi-VR terminal collaborative interaction method based on mutual positioning calculation, the multi-VR terminal collaborative interaction method comprising the following steps: Multiple VR terminals can locate each other by exchanging visual and ranging data, thereby building a collaborative perception network; Each VR terminal acts as an observer, and performs pose confidence assessment on the virtual character mapped by each VR terminal in the collaborative perception network to obtain the observation confidence of the pose of each virtual character. When all VR terminals detect the presence of a virtual character with lost visual observation, the last observation information of the lost virtual character is broadcast to the collaborative perception network. Based on all the last observation information and all the observation confidence, the pose constraint boundary of the lost virtual character during the VR terminal occlusion interruption is determined. Trajectory prediction is performed on the skeletal key points of the lost virtual character to obtain the motion prediction trajectory of the skeletal key points of the virtual character. Based on the pose constraint boundary and the motion prediction trajectory, the virtual character whose observation is lost is subjected to pose transition, and the pose transition parameters of the virtual character are obtained. Then, the spatial position of the virtual character is adjusted through the pose transition parameters.

[0006] In this embodiment, the mutual positioning of multiple VR terminals by exchanging visual data and ranging data specifically includes: Acquire visual and ranging data collected by each VR terminal; The visual data and ranging data collected by each VR terminal are encapsulated into a positioning data packet; All location data packets are synchronously exchanged between various VR terminals at a preset frequency; The relative pose relationships between each VR terminal are determined based on the positioning data packets obtained from the exchange.

[0007] In this embodiment, constructing a collaborative sensing network specifically includes: Obtain the relative pose relationships between various VR terminals; A collaborative perception network is constructed based on the relative pose relationships between various VR terminals.

[0008] In this embodiment, each VR terminal acts as an observer, performing pose confidence assessment on the virtual character mapped to each VR terminal in the collaborative perception network. The observation confidence of each virtual character's pose specifically includes: Each VR terminal captures the virtual characters mapped by other VR terminals in real time and obtains the observed pose of each virtual character; Determine the feature point matching degree and observation distance factor for the observation pose of each virtual character; Determine the degree of occlusion and environmental complexity of the observation pose for each virtual character; The observation reliability of each virtual character's pose is determined based on the feature point matching degree, observation distance factor, occlusion degree, and environmental complexity.

[0009] In this embodiment, determining the pose constraint boundaries of the virtual character whose observations were lost during the VR terminal occlusion interruption, based on all the last observation information and all observation confidence levels, specifically includes: Obtain the last observation information about the lost virtual character broadcast by all VR terminals, specifically including the last observation pose and the last observation timestamp; Based on all the last observation information and all the observation confidence, determine the reference position and reference pose of the virtual character that was lost during the VR terminal occlusion interruption. The position uncertainty ellipsoid and attitude uncertainty range of the virtual character lost during the VR terminal occlusion interruption are determined by the reference position and the reference attitude. The pose constraint boundaries of the virtual character whose observations were lost during the VR terminal occlusion interruption are determined based on the position uncertainty ellipsoid and the pose uncertainty range.

[0010] In this embodiment, trajectory prediction is performed on the skeletal keypoints of the lost virtual character to obtain the motion prediction trajectory of the skeletal keypoints of the virtual character. Specifically, this includes: The motion sequence of the virtual character whose observation was lost was obtained before the VR terminal occlusion interrupted, and the motion sequence included the position, velocity and acceleration of the skeletal key points; The motion sequence is used to predict the short-term trajectory of the skeletal key points of the virtual character in the early stage of occlusion, and the short-term predicted trajectory of the skeletal key points of the virtual character is obtained. The motion sequence is used to predict the long-term trajectory of the skeletal key points of the virtual character in the later stages of occlusion, and the long-term predicted trajectory of the skeletal key points of the virtual character is obtained. The short-term predicted trajectory and the long-term predicted trajectory are fused to obtain the motion prediction trajectory of the skeletal key points of the virtual character.

[0011] In this embodiment, the virtual character whose observations have been lost is subjected to a pose transition based on the pose constraint boundary and the motion prediction trajectory. The pose transition parameters of the virtual character specifically include: The pose transition of the virtual character with lost observation is optimized by using the pose constraint boundary and the motion prediction trajectory to obtain the position transition sequence of the root bone of the virtual character during the pose transition. The rotational transition sequence of the skeletal key points of the virtual character is determined based on the positional transition sequence; The pose transition parameters of the virtual character are determined by the position transition sequence and the rotation transition sequence.

[0012] In this embodiment, adjusting the spatial position of the virtual character through the posture transition parameters specifically includes: The virtual character's transitional display pose for each frame during the occlusion interruption is generated based on the posture transition parameters. When visual observation is restored, the virtual character's spatial position is updated using the virtual character's real-time pose and all transitional display poses.

[0013] In this embodiment, the posture transition parameter represents a set of data that continuously drives the smooth movement of the virtual character during the VR terminal occlusion interruption.

[0014] Secondly, this application provides a multi-VR terminal collaborative interaction system based on mutual positioning calculation, used to execute a multi-VR terminal collaborative interaction method based on mutual positioning calculation, the multi-VR terminal collaborative interaction system comprising: The mutual positioning module is used by multiple VR terminals to locate each other by exchanging visual data and ranging data, thereby building a collaborative perception network; The credibility assessment module is used by each VR terminal as an observer to assess the pose credibility of the virtual character mapped by each VR terminal in the collaborative perception network, and obtain the observation credibility of the pose of each virtual character. The pose constraint module is used to broadcast the last observation information of the virtual character whose visual observation is lost to the cooperative perception network when all VR terminals detect the existence of a virtual character whose visual observation is lost, and to determine the pose constraint boundary of the virtual character whose observation is lost during the VR terminal occlusion interruption based on all the last observation information and all the observation confidence. The trajectory prediction module is used to predict the trajectory of the skeletal key points of the virtual character that have been lost in observation, and to obtain the motion prediction trajectory of the skeletal key points of the virtual character. The posture transition module is used to perform posture transition on the virtual character whose observation has been lost according to the posture constraint boundary and the motion prediction trajectory, to obtain the posture transition parameters of the virtual character, and then adjust the spatial position of the virtual character through the posture transition parameters.

[0015] The technical solutions provided by the embodiments disclosed in this application have the following beneficial effects: Multiple VR terminals mutually locate each other by exchanging visual and ranging data, thereby constructing a collaborative perception network. Each VR terminal acts as an observer, evaluating the pose reliability of the virtual character mapped to each VR terminal in the collaborative perception network to obtain the observation reliability of each virtual character's pose. When all VR terminals detect a virtual character with lost visual observation, they broadcast the last observation information of the lost virtual character to the collaborative perception network. Based on all the last observation information and all the observation reliability, the pose constraint boundary of the lost virtual character during the VR terminal occlusion interruption is determined. The trajectory of the skeletal keypoints of the lost virtual character is predicted to obtain the motion prediction trajectory of the virtual character's skeletal keypoints. The pose transition of the lost virtual character is performed according to the pose constraint boundary and the motion prediction trajectory to obtain the virtual character's pose transition parameters, and then the spatial position of the virtual character is adjusted through the pose transition parameters.

[0016] Therefore, it can be seen that in this application, the spatial position of the virtual character can be adjusted through the posture transition parameters. Firstly, by constructing a collaborative perception network through the exchange of visual and ranging data from multiple VR terminals, the limitations of existing technologies relying on single-terminal visual observation are overcome, providing multi-source data support for subsequent pose processing and effectively avoiding the data loss problem caused by single-terminal occlusion. Secondly, each VR terminal performs a credibility assessment of the virtual character's pose, which can filter out pose data with higher observation quality and eliminate interference from low-credibility data, ensuring that the pose information before occlusion truly reflects the actual motion state of the virtual character. Furthermore, when all VR terminals detect a virtual character with lost visual observation, the pose constraint boundary is determined by broadcasting the last observation information and combining it with the observation credibility. This pose constraint boundary is based on multi-terminal data. The overall results not only avoid the rigid illusion of "freezing" characters in existing technologies, but also prevent excessive pose deviations caused by unconstrained prediction. It limits the reasonable movement range of virtual characters in terms of spatial scope, reducing the risk of abrupt changes during visual recovery. Furthermore, trajectory prediction of key skeletal points of virtual characters generates motion prediction trajectories that conform to the continuity and correlation of human movement, which are much closer to real movements than traditional linear interpolation, solving the problem of the disconnect between traditional prediction methods and actual motion trajectories. Finally, by combining pose constraint boundaries and motion prediction trajectories to perform posture transitions and adjust the spatial position of virtual characters, a seamless transition of pose from the occlusion period to the visual recovery is achieved, eliminating abrupt changes and spatial misalignment, ensuring the consistency of the virtual world state from the perspective of all participating terminals, and significantly improving the immersive experience of multi-user collaboration.

[0017] In summary, the technical solution adopted in this application can prevent virtual characters from experiencing visual jumps when the VR terminal is interrupted by occlusion. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only for this embodiment of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is an exemplary flowchart of a multi-VR terminal collaborative interaction method based on mutual positioning calculation provided in this application; Figure 2 This is a flowchart illustrating the process of determining pose constraint boundaries according to the present application. Figure 3 This is a flowchart illustrating the process of determining the predicted motion trajectory provided in this application; Figure 4 This is a module structure diagram of a multi-VR terminal collaborative interaction system based on mutual positioning calculation provided in this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] This application provides a multi-VR terminal collaborative interaction system and method based on mutual positioning calculation. The core of this system involves multiple VR terminals exchanging visual and ranging data to mutually locate each other, thereby constructing a collaborative perception network. Each VR terminal acts as an observer, evaluating the pose reliability of the virtual character mapped to each VR terminal in the collaborative perception network to obtain the observation reliability of each virtual character's pose. When all VR terminals detect a virtual character with lost visual observation, they broadcast the last observation information of the lost virtual character to the collaborative perception network. Based on all the last observation information and all the observation reliability, the pose constraint boundary of the lost virtual character during the VR terminal occlusion interruption is determined. Trajectory prediction is performed on the skeletal keypoints of the lost virtual character to obtain the predicted motion trajectory of the virtual character's skeletal keypoints. The posture transition of the lost virtual character is performed based on the pose constraint boundary and the predicted motion trajectory to obtain the virtual character's posture transition parameters, which are then used to adjust the virtual character's spatial position.

[0022] Example 1: To better understand the above technical solution, the following will provide a detailed description of the technical solution in conjunction with the accompanying drawings and specific implementation methods. (Refer to...) Figure 1 As shown in the figure, this is an exemplary flowchart of a multi-VR terminal collaborative interaction method based on mutual positioning calculation according to this embodiment of the application. The multi-VR terminal collaborative interaction method includes the following steps: In step S1, multiple VR terminals locate each other by exchanging visual data and ranging data, thereby constructing a collaborative perception network.

[0023] In this embodiment, the mutual positioning of multiple VR terminals by exchanging visual data and ranging data can be achieved through the following steps: Acquire visual and ranging data collected by each VR terminal; The visual data and ranging data collected by each VR terminal are encapsulated into a positioning data packet; All location data packets are synchronously exchanged between various VR terminals at a preset frequency; The relative pose relationships between each VR terminal are determined based on the positioning data packets obtained from the exchange.

[0024] It should be noted that the visual data mentioned in this application represents image data of the environment surrounding the VR terminal; the ranging data represents distance information data between the VR terminal and surrounding objects; the positioning data packet represents a data structure that encapsulates visual features and distance measurement information; and the relative pose relationship represents the relative position and posture transformation relationship between two VR terminals.

[0025] In practice, firstly, each VR terminal acquires color images at 30 frames per second using its built-in RGB camera and simultaneously acquires corresponding depth images using an infrared depth sensor. These two types of images are used as visual data. Simultaneously, the built-in six-axis inertial measurement unit acquires raw data containing three-axis acceleration and three-axis angular velocity at a frequency of 100 Hz. This raw data is then low-pass filtered to remove high-frequency noise and used as ranging data. Each set of visual and ranging data is stamped with a microsecond-level timestamp from the terminal's local clock. Secondly, the acquired color images are processed using the ORB feature extraction algorithm to extract at least 200 scale-invariant feature points and their 256-bit binary descriptors. The pixel coordinates of these feature points, their corresponding depth values, and their descriptor binary codes, along with the filtered acceleration and angular velocity data acquired at the same time, the timestamp, and the terminal's unique identifier, are encapsulated into a positioning data packet according to a custom binary protocol. The header of this positioning data packet contains the data length and checksum. Finally, all VR terminals support Wi-Fi... A decentralized peer-to-peer network is built using a wireless network card based on Direct technology. Each VR terminal starts a timer to broadcast encapsulated location data packets to all other VR terminals in the peer-to-peer network at a preset frequency of 10 times per second via the User Datagram Protocol (UDP). Simultaneously, each VR terminal listens for and receives location data packets from other VR terminals, using Network Time Protocol (NTP) to synchronize the timestamps of all terminals, ensuring a time deviation of less than 1 millisecond. Finally, for each received location data packet from another VR terminal, Hamming distance is used for feature descriptor matching to find at least 50 pairs of matching feature points. Then, combining the 3D coordinates of these feature points in the depth image and their corresponding 3D coordinates in the peer's location data packet, a random sampling consensus algorithm is used to eliminate incorrect matching pairs. Using the 3D coordinates of the remaining correct matching pairs, a 3x3 rotation matrix and a 3x1 translation vector are calculated using singular value decomposition (SVD). The homogeneous transformation matrix formed by the rotation matrix and translation vector is used as the relative pose relationship between the VR terminal and the peer VR terminal, thus obtaining the relative pose relationship between all VR terminals.

[0026] In this embodiment, the cooperative sensing network can be constructed using the following steps: Obtain the relative pose relationships between various VR terminals; A collaborative perception network is constructed based on the relative pose relationships between various VR terminals.

[0027] It should be noted that the distributed network described in this application involves multiple VR terminals collaboratively sharing virtual character pose information.

[0028] In specific implementation, firstly, the relative pose relationships between each VR terminal are obtained; secondly, a VR terminal is selected as the spatial reference origin, all VR terminals are treated as nodes, and the relative pose relationship between any two terminals is used as a directed edge connecting these two nodes, constructing a directed graph data structure. A breadth-first search algorithm is used to traverse this directed graph. For each node in the directed graph except the origin, matrix multiplication is performed sequentially on all homogeneous transformation matrices along the path from the origin to that node, and the result of the matrix multiplication is used as the global pose of the corresponding VR terminal in the reference origin coordinate system. The calculated global poses of all terminals are bound to their unique identifiers to form a global pose mapping table. Then, based on this global pose mapping table, a local state database is initialized on each VR terminal, storing its own state and that of all other VR terminals. The real-time status of the VR terminal includes the terminal identifier, global pose, timestamp, and the latest pose data of the virtual character mapped to the VR terminal. Simultaneously, a network synchronization daemon is started on each VR terminal. This daemon periodically (e.g., 10 times per second) encapsulates its complete status data into data packets and sends them via UDP multicast to a preset multicast group address. It continuously listens to this address to receive similar data packets from other VR terminals. Upon receiving a data packet, it parses out the VR terminal identifier, timestamp, and status data. If the timestamp is newer than the timestamp recorded for the corresponding VR terminal identifier in the local status database, the new data overwrites the old data to update the local status database. Finally, the overall distributed network formed by the local status databases running on all VR terminals and their network synchronization daemons constitutes a cooperative perception network.

[0029] In step S2, each VR terminal acts as an observer, and performs pose confidence assessment on the virtual character mapped by each VR terminal in the collaborative perception network to obtain the observation confidence of the pose of each virtual character.

[0030] In this embodiment, each VR terminal acts as an observer, and performs pose confidence assessment on the virtual character mapped by each VR terminal in the cooperative perception network. The observation confidence of the pose of each virtual character can be obtained by the following steps: Each VR terminal captures the virtual characters mapped by other VR terminals in real time and obtains the observed pose of each virtual character; Determine the feature point matching degree and observation distance factor for the observation pose of each virtual character; Determine the degree of occlusion and environmental complexity of the observation pose for each virtual character; The observation reliability of each virtual character's pose is determined based on the feature point matching degree, observation distance factor, occlusion degree, and environmental complexity.

[0031] It should be noted that, in this application, the observed pose refers to the position and posture of the virtual character at the current moment; the feature point matching degree refers to the degree of overlap between the human feature points actually captured by the VR terminal and the feature points of the virtual character obtained by projection based on the observed pose; the observation distance factor refers to the attenuation factor that affects the observation quality; the occlusion degree refers to the proportion of the virtual character being occluded by environmental objects in the VR terminal's field of view; the environmental complexity refers to the degree of clutter and dynamic change of the background in the VR terminal's field of view; and the observation reliability refers to the parameter value for evaluating the accuracy of the observed pose of the virtual character.

[0032] In practice, firstly, each VR terminal receives the observed poses of its virtual character from other VR terminals in real time through a collaborative sensing network. Secondly, for each VR terminal, its local RGB camera is activated to capture environmental images at a rate of 30 frames per second, and the MediaPipe human pose estimation algorithm is used to detect 2D human keypoints from the images. The received observed poses of the virtual character are then transformed into a 2D image plane using the camera projection model on the VR terminal itself, resulting in a set of projected keypoint coordinates. The keypoint coordinates detected by MediaPipe are matched with the projected keypoint coordinates using nearest neighbor matching, and the average Euclidean distance between all matching point pairs is calculated. The reciprocal of this distance is used as the feature point matching degree of the virtual character's observed pose mapped by the VR terminal, thus obtaining the feature point matching degree of each virtual character's observed pose. Simultaneously, the position of the VR terminal itself in the reference origin coordinate system and the received virtual character's pose are calculated. The three-dimensional Euclidean distance of the position in the character's observed pose is mapped to a value between 0 and 1 using a sigmoid function, thus obtaining the observation distance factor of each virtual character's observed pose. Secondly, for each VR terminal, the local depth camera is activated to acquire a depth image. The depth value corresponding to the projection keypoint region of the virtual character mapped by the VR terminal is compared with the depth value of the corresponding pixel in the depth image. The proportion of pixels with depth values ​​greater than the image depth value plus a threshold is counted, and this proportion is used as the occlusion degree of the virtual character's observed pose mapped by the VR terminal. Simultaneously, the entropy value of the image surrounding the virtual character's projection area is calculated, and the entropy value is normalized to a value between 0 and 1 as the environmental complexity of the virtual character's observed pose mapped by the VR terminal, thus obtaining the occlusion degree and environmental complexity of each virtual character's observed pose.Next, for each virtual character, a weighting method based on the coefficient of variation is used to calculate the weights of four parameters: feature point matching degree, observation distance factor, occlusion degree, and environmental complexity. Specifically, all historical data of the virtual character observed by the VR terminal in the past 30 seconds are collected. Each data point includes the feature point matching degree, observation distance factor, occlusion degree, and environmental complexity at that moment, forming a four-column data matrix. The mean and standard deviation of each column are calculated, and then the coefficient of variation (standard deviation divided by mean) of each column is calculated. The reciprocal of the coefficient of variation of each column is then calculated, and the four reciprocals are normalized (so that their sum is 1). The normalized results are used as the weights w1 for feature point matching degree, w2 for observation distance factor, w3 for occlusion degree, and w4 for environmental complexity. Then, the feature point matching degree at the current moment is multiplied by w1, the observation distance factor by w2, the occlusion degree by w3, and the environmental complexity by w4. These four products are then substituted into the formula to calculate the weighted sum = w1 * feature point matching degree + The weighted sum is calculated as w2*observation distance factor - w3*occlusion degree - w4*environment complexity. This weighted sum is then mapped to a value between 0 and 1 using the Sigmoid function. The final mapped value is used as the observation confidence level for each virtual character's pose.

[0033] In step S3, when all VR terminals detect the presence of a virtual character with lost visual observation, the last observation information of the lost virtual character is broadcast to the collaborative perception network. Based on all the last observation information and all the observation confidence, the pose constraint boundary of the lost virtual character during the VR terminal occlusion interruption is determined.

[0034] In practice, when all VR terminals detect a virtual character with lost visual observation, they broadcast the last observation information of the lost virtual character to the collaborative perception network.

[0035] Preferably, in this embodiment, the pose constraint boundaries of the virtual character whose observations were lost during the VR terminal occlusion interruption are determined based on all the last observation information and all observation confidence levels, with reference to... Figure 2 As shown in the figure, this is a flowchart illustrating the process of determining pose constraint boundaries in some embodiments of this application. In this embodiment, determining pose constraint boundaries can be achieved using the following steps: In step S31, the last observation information about the lost virtual character broadcast by all VR terminals is obtained. The last observation information specifically includes the last observation pose and the last observation timestamp. In step S32, the reference position and reference pose of the virtual character whose observation was lost during the VR terminal occlusion interruption are determined based on all the last observation information and all the observation confidence. In step S33, the position uncertainty ellipsoid and attitude uncertainty range of the virtual character whose observation was lost during the VR terminal occlusion interruption are determined by the reference position and the reference attitude; In step S34, the pose constraint boundary of the virtual character whose observation was lost during the VR terminal occlusion interruption is determined based on the position uncertainty ellipsoid and the pose uncertainty range.

[0036] It should be noted that the position uncertainty ellipsoid mentioned in this application represents a three-dimensional ellipsoidal space region centered on the position of the reference pose, and the position of the virtual character during the VR terminal occlusion interruption is considered to fall within this region with a certain probability; the pose uncertainty range represents the possible angular range of the virtual character's pose deviating from the reference pose during the VR terminal occlusion interruption; and the pose constraint boundary represents the boundary of the pose change of the virtual character with lost observations during the VR terminal occlusion interruption.

[0037] In practice, firstly, the last observation information for the same lost observation virtual character broadcast by all VR terminals is obtained. This last observation information specifically includes the last observed pose (containing position vectors and pose quaternions) and the last observed timestamp. Secondly, all last observed timestamps are aligned to the earliest timestamp. Linear interpolation is then used to linearly interpolate the position vectors from other times to the earliest timestamp. Then, using the observation confidence level corresponding to each linearly interpolated position vector as a weight, a weighted average is calculated for all position vectors. The resulting weighted average position is then taken as the lost observation position. The baseline position of the virtual character during the VR terminal occlusion interruption was determined. Simultaneously, spherical linear interpolation was used to linearly interpolate the pose quaternions at other times to the earliest timestamp. Then, using the observation confidence level corresponding to each linearly interpolated pose quaternion as a weight, a weighted average was calculated for all pose quaternions. This weighted average quaternion was used as the baseline pose of the virtual character during the VR terminal occlusion interruption. Next, using all linearly interpolated position vectors as sample points and their corresponding observation confidence levels as weights, a weighted covariance matrix was calculated. Singular value decomposition (SVD) is performed to obtain three mutually orthogonal eigenvectors and their corresponding eigenvalues. These three eigenvectors are used as the three principal axes of an ellipsoid, and the square root of each eigenvalue is used as the semi-axis length of the corresponding principal axis. This constructs an ellipsoid centered at the reference position, with the eigenvectors as the directions and the semi-axis lengths as the dimensions. This ellipsoid is then used as the position uncertainty ellipsoid for the lost virtual character during the VR terminal occlusion interruption period. Simultaneously, the angle difference between each linearly interpolated pose quaternion and the reference pose is calculated, and the standard deviation of these angle differences is calculated. The difference is used as the half-angle width to construct a conical range centered on the reference attitude axis and with the half-angle width as the half-apex angle. This conical range is used as the attitude uncertainty range of the virtual character whose observation is lost during the VR terminal occlusion interruption. Finally, the three-dimensional spatial position range described by the position uncertainty ellipsoid and the attitude rotation angle range described by the attitude uncertainty range are combined to form the pose boundary of the virtual character during the VR terminal occlusion interruption. The obtained pose boundary is used as the pose constraint boundary of the virtual character whose observation is lost during the VR terminal occlusion interruption.

[0038] In step S4, trajectory prediction is performed on the skeletal key points of the lost virtual character to obtain the motion prediction trajectory of the skeletal key points of the virtual character.

[0039] Preferably, in this embodiment, trajectory prediction is performed on the skeletal key points of the lost virtual character to obtain the predicted motion trajectory of the skeletal key points of the virtual character, with reference to... Figure 3As shown in the figure, this is a flowchart illustrating the process of determining the motion prediction trajectory in some embodiments of this application. In this embodiment, the determination of the motion prediction trajectory can be achieved by the following steps: In step S41, the motion sequence of the virtual character whose observation was lost before the VR terminal occlusion interrupted is obtained, and the motion sequence includes the position, velocity and acceleration of the skeletal key points; In step S42, the motion sequence is used to predict the short-term trajectory of the skeletal key points of the virtual character in the early stage of occlusion, so as to obtain the short-term predicted trajectory of the skeletal key points of the virtual character. In step S43, the long-term trajectory prediction of the skeletal key points of the virtual character in the later stage of occlusion is performed by the motion sequence to obtain the long-term predicted trajectory of the skeletal key points of the virtual character. In step S44, the short-term predicted trajectory and the long-term predicted trajectory are fused to obtain the motion prediction trajectory of the skeletal key points of the virtual character.

[0040] It should be noted that the short-term predicted trajectory mentioned in this application refers to the predicted trajectory of the skeletal key points of the virtual character in the initial period after the VR terminal occlusion is interrupted; the long-term predicted trajectory refers to the predicted trajectory of the skeletal key points of the virtual character in a longer period after the VR terminal occlusion is interrupted; and the motion predicted trajectory refers to the predicted trajectory of the motion of the skeletal key points of the virtual character after the VR terminal occlusion is interrupted.

[0041] In specific implementation, firstly, the historical data of the lost virtual character in the most recent N frames (e.g., N=30, corresponding to approximately 0.5 seconds) is acquired. For each skeletal keypoint in each frame, its instantaneous velocity is calculated using the position difference between adjacent frames, and its instantaneous acceleration is calculated using the velocity difference between adjacent frames. The position, velocity, and acceleration are arranged in chronological order to form a motion sequence. Secondly, a uniformly accelerated motion model based on a Kalman filter is adopted. The position, velocity, and acceleration of each skeletal keypoint at the last time point in the motion sequence are used as the initial state of the Kalman filter. Then, the time update step of the Kalman filter is run to predict the position of each skeletal keypoint at each moment in the next 0.3 seconds at 0.01-second intervals. These predicted positions are connected in chronological order, and the resulting trajectory is used as the short-term predicted trajectory of the virtual character's skeletal keypoints. Next, an autoregressive integral moving average model is adopted. The position history of each skeletal keypoint in the motion sequence is used as a one-dimensional time series input. The model parameters are fitted using the least squares method, and then the fitted model is used to iteratively predict the motion sequence. The positions of key points on the skeleton are predicted at 0.1-second intervals within the next 2.0 seconds. These predicted positions are then connected in chronological order, and the resulting trajectory is used as the long-term predicted trajectory of the virtual character's key points. Finally, a piecewise cubic spline curve is used to smoothly connect the short-term and long-term predicted trajectories. Specifically, the last point of the short-term predicted trajectory (t=0.3 seconds) and the nearest point of the long-term predicted trajectory after t=0.3 seconds (t=0.4 seconds) are set as the two fixed endpoints of the spline curve. From t=0.3 seconds to t... Within a time interval of 0.4 seconds, a new smooth transition trajectory is generated using a cubic spline interpolation algorithm, resulting in a spline curve. This spline curve replaces the original data points of the long-term predicted trajectory between t=0.3 seconds and t=0.4 seconds. The short-term predicted trajectory (t=0-0.3 seconds), this spline curve (t=0.3-0.4 seconds), and the remaining part of the long-term predicted trajectory (t>0.4 seconds) are then connected in chronological order to form a complete trajectory. This complete trajectory is then used as the motion prediction trajectory for the skeletal key points of the virtual character.

[0042] In step S5, the virtual character whose observation was lost is subjected to posture transition according to the pose constraint boundary and the motion prediction trajectory to obtain the posture transition parameters of the virtual character, and then the spatial position of the virtual character is adjusted through the posture transition parameters.

[0043] In this embodiment, the virtual character whose observations have been lost is subjected to a pose transition based on the pose constraint boundary and the motion prediction trajectory. The pose transition parameters of the virtual character can be obtained by the following steps: The pose transition of the virtual character with lost observation is optimized by using the pose constraint boundary and the motion prediction trajectory to obtain the position transition sequence of the root bone of the virtual character during the pose transition. The rotational transition sequence of the skeletal key points of the virtual character is determined based on the positional transition sequence; The pose transition parameters of the virtual character are determined by the position transition sequence and the rotation transition sequence.

[0044] It should be noted that the position transition sequence mentioned in this application refers to the sequence formed by arranging the three-dimensional position coordinates of the root bone of the virtual character at each discrete time point in chronological order during the estimated occlusion interruption; the rotation transition sequence is the sequence formed by arranging the rotation quaternions of each non-root bone of the virtual character at each discrete time point in chronological order during the estimated occlusion interruption; and the posture transition parameters represent the data set that continuously drives the smooth movement of the virtual character during the VR terminal occlusion interruption.

[0045] In specific implementation, firstly, the position coordinates of the root skeleton at all prediction time points are obtained from the motion prediction trajectory as the desired reference trajectory. Simultaneously, the center, principal axis direction, semi-axis length, and reference axis and half-angle width of the position uncertainty ellipsoid are obtained from the pose constraint boundary. Secondly, a transition optimization model for the root skeleton position is established. The decision variables of this transition optimization model are the three-dimensional position of the root skeleton at each moment during occlusion. The optimization objective function is set to minimize the sum of two parts: the first part is the sum of the squares of the position differences between the sequence of decision variables and the desired reference trajectory for all corresponding points; the second part is the sum of the squares of the position differences between adjacent points in the decision variable sequence. The sum of squares of the changes in position (i.e., velocity) at each time point is used to ensure smooth motion. The constraints of this transition optimization model are set as follows: at each time step, the position represented by the decision variable must be located within the ellipsoidal space centered on the center of the ellipsoid of position uncertainty at that time, with its principal axis direction and semi-axis length as parameters. Then, a sequential quadratic programming algorithm is used to solve the above constrained optimization problem. Initially, the desired reference trajectory is used as the initial solution. The optimal solution is approximated by iteratively solving a series of quadratic programming subproblems. The optimal positions of the root skeletons that satisfy all constraints at each time point are arranged in chronological order, and this sequence is used as the root skeleton of the virtual character. The system generates a position transition sequence of the skeleton during posture transitions. Then, for each time point in the position transition sequence, the rotation quaternion of all non-root bones at that moment is calculated using an inverse kinematics algorithm. Specifically, using the root bone position at that moment and the target positions of other bone key points obtained from the motion prediction trajectory as input, an iterative algorithm based on Jacobi transpose is used to calculate the rotation angle of each joint starting from the root bone, so that the position of each end effector (e.g., palm, foot) is as close as possible to its corresponding target position. In each iteration, the adjustment amount of joint rotation is calculated using the Jacobi transpose matrix based on the error between the current position and the target position. The rotation angle is limited to its physiological range of motion. The rotation quaternions of each skeletal key point calculated at each time point are recorded and then arranged in chronological order. The resulting sequence is used as the rotation transition sequence of the skeletal key points of the virtual character. Finally, the position transition sequence and the rotation transition sequence are aligned along the same time axis. A continuous and smooth time function curve is generated for each sequence using a cubic spline interpolation algorithm. These two spline curves are used as the position spline curve and the rotation spline curve, respectively. The position spline curve and the rotation spline curve are then combined into a structured parameter, which is used as the posture transition parameter of the virtual character.

[0046] In this embodiment, adjusting the spatial position of the virtual character using the posture transition parameters can be achieved through the following steps: The virtual character's transitional display pose for each frame during the VR terminal's occlusion interruption is generated based on the aforementioned pose transition parameters. When visual observation is restored, the virtual character's spatial position is updated using the virtual character's real-time pose and all transitional display poses.

[0047] In specific implementation, firstly, after the VR terminal enters the occlusion interruption state for the virtual character, its graphics rendering engine starts a transition rendering thread. At the beginning of each frame rendering cycle of this thread, the precise timestamp of the current frame is obtained. Based on the timestamp, the corresponding 3D position coordinates and rotation quaternions are obtained on the position spline curve and rotation spline curve included in the pose transition parameters. The calculated 3D position coordinates and rotation quaternions are combined into a homogeneous transformation matrix, and this homogeneous transformation matrix is ​​used as the transition display pose of the current frame and submitted to the rendering pipeline to update the world transformation matrix of the virtual character, thereby realizing its driving display in virtual space. Secondly, when a network message indicating that the virtual character has been re-observed and its attached real-time pose data are received from any other VR terminal through the cooperative perception network, the virtual character is determined to have recovered visual observation. At this time, the last generated transition display pose and its corresponding timestamp T_end are immediately recorded, and a continuous K-frame (e.g., K=15, corresponding to The smooth transition process (approximately 0.25 seconds) involves calculating a blending coefficient α that linearly increases from 0 to 1 in each frame, based on the time difference between the current timestamp and T_end. The blending coefficient α = time difference between the current timestamp and T_end / total time of K frames. For the root bone position of the virtual character, a linear interpolation algorithm is used to interpolate between the position in the last transitional display pose and the position in the real-time pose. For the rotation of each bone keypoint of the virtual character, a spherical linear interpolation algorithm is used to interpolate between the rotation quaternions in the last transitional display pose and the rotation quaternions in the real-time pose. The weights of both interpolations are controlled by the blending coefficient α. The new position and the new rotation quaternion array obtained from the interpolation are combined to form a new pose, which is then used as the updated spatial position of the virtual character for rendering in that frame. After the smooth transition process has completed K frames, the blending coefficient α reaches 1, and the spatial position of the virtual character will be completely driven by the real-time pose, thus completing the spatial position update from the predicted transition state to the actual observation state.

[0048] Therefore, it can be seen that in this application, the spatial position of the virtual character can be adjusted through the posture transition parameters. Firstly, by constructing a collaborative perception network through the exchange of visual and ranging data from multiple VR terminals, the limitations of existing technologies relying on single-terminal visual observation are overcome, providing multi-source data support for subsequent pose processing and effectively avoiding the data loss problem caused by single-terminal occlusion. Secondly, each VR terminal performs a credibility assessment of the virtual character's pose, which can filter out pose data with higher observation quality and eliminate interference from low-credibility data, ensuring that the pose information before occlusion truly reflects the actual motion state of the virtual character. Furthermore, when all VR terminals detect a virtual character with lost visual observation, the pose constraint boundary is determined by broadcasting the last observation information and combining it with the observation credibility. This pose constraint boundary is a comprehensive result of multi-terminal data. The combined results not only avoid the rigid illusion of "freezing" characters in existing technologies, but also prevent excessive deviation in pose caused by unconstrained prediction. It limits the reasonable movement range of virtual characters in terms of spatial scope, reducing the risk of abrupt changes during visual recovery. Furthermore, trajectory prediction of key skeletal points of virtual characters conforms to the continuity and correlation of human movement. The generated motion prediction trajectory is much closer to real movements than traditional linear interpolation, solving the problem of the disconnect between traditional prediction methods and actual motion trajectories. Finally, by combining pose constraint boundaries and motion prediction trajectories to perform posture transitions and adjust the spatial position of virtual characters, a seamless connection between pose during occlusion and after visual recovery is achieved. This completely eliminates abrupt changes and spatial misalignment, ensuring the consistency of the virtual world state from the perspective of all participating terminals and significantly improving the immersive experience of multi-user collaboration.

[0049] In summary, the technical solution adopted in this application can prevent virtual characters from experiencing visual jumps when the VR terminal is interrupted by occlusion.

[0050] Example 2: This application provides a multi-VR terminal collaborative interaction system based on mutual positioning calculation, referencing... Figure 4 As shown in the figure, this is a block structure diagram of a multi-VR terminal collaborative interaction system based on mutual positioning calculation according to this embodiment of the present application. The multi-VR terminal collaborative interaction system includes: The mutual positioning module 100 is used for multiple VR terminals to locate each other by exchanging visual data and ranging data, thereby building a collaborative perception network; The credibility assessment module 200 is used by each VR terminal as an observer to perform pose credibility assessment on the virtual character mapped by each VR terminal in the collaborative perception network, and obtain the observation credibility of the pose of each virtual character. The pose constraint module 300 is used to broadcast the last observation information of the virtual character whose visual observation is lost to the collaborative perception network when all VR terminals detect the existence of a virtual character whose visual observation is lost, and to determine the pose constraint boundary of the virtual character whose observation is lost during the occlusion interruption period of the VR terminal based on all the last observation information and all the observation confidence. The trajectory prediction module 400 is used to predict the trajectory of the skeletal key points of the virtual character whose observation is lost, and to obtain the motion prediction trajectory of the skeletal key points of the virtual character. The attitude transition module 500 is used to perform attitude transition on the virtual character whose observation has been lost according to the pose constraint boundary and the motion prediction trajectory, to obtain the attitude transition parameters of the virtual character, and then adjust the spatial position of the virtual character through the attitude transition parameters.

[0051] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0052] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compactdisc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0053] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

Claims

1. A multi-VR terminal cooperative interaction method based on mutual positioning calculation, characterized in that, The multi-VR terminal collaborative interaction method includes the following steps: Multiple VR terminals can locate each other by exchanging visual and ranging data, thereby building a collaborative perception network; Each VR terminal acts as an observer, and performs pose confidence assessment on the virtual character mapped by each VR terminal in the collaborative perception network to obtain the observation confidence of the pose of each virtual character. In this context, each VR terminal acts as an observer, evaluating the pose reliability of the virtual character mapped to each VR terminal in the collaborative perception network. The observation reliability of each virtual character's pose specifically includes: Each VR terminal captures the virtual characters mapped by other VR terminals in real time and obtains the observed pose of each virtual character; Determine the feature point matching degree and observation distance factor for the observation pose of each virtual character; Determine the degree of occlusion and environmental complexity of the observation pose for each virtual character; The observation reliability of each virtual character's pose is determined based on the feature point matching degree, observation distance factor, occlusion degree, and environmental complexity. When all VR terminals detect the presence of a virtual character with lost visual observation, the last observation information of the lost virtual character is broadcast to the collaborative perception network. Based on all the last observation information and all the observation confidence, the pose constraint boundary of the lost virtual character during the VR terminal occlusion interruption is determined. Specifically, determining the pose constraint boundaries of the virtual character during the VR terminal occlusion interruption based on all the last observation information and all observation confidence levels includes: Obtain the last observation information about the lost virtual character broadcast by all VR terminals, specifically including the last observation pose and the last observation timestamp; Based on all the last observation information and all the observation confidence, determine the reference position and reference pose of the virtual character that was lost during the VR terminal occlusion interruption. The position uncertainty ellipsoid and attitude uncertainty range of the virtual character lost during the VR terminal occlusion interruption are determined by the reference position and the reference attitude. The pose constraint boundaries of the virtual character whose observations were lost during the VR terminal occlusion interruption are determined based on the position uncertainty ellipsoid and the pose uncertainty range. Trajectory prediction is performed on the skeletal key points of the lost virtual character to obtain the motion prediction trajectory of the skeletal key points of the virtual character. Based on the pose constraint boundary and the motion prediction trajectory, the virtual character whose observation is lost is subjected to pose transition, and the pose transition parameters of the virtual character are obtained. Then, the spatial position of the virtual character is adjusted through the pose transition parameters.

2. The multi-VR terminal cooperative interaction method based on mutual positioning calculation of claim 1, wherein, Multiple VR terminals perform mutual positioning by exchanging visual and ranging data, specifically including: Acquire visual and ranging data collected by each VR terminal; The visual data and ranging data collected by each VR terminal are encapsulated into a positioning data packet; All location data packets are synchronously exchanged between various VR terminals at a preset frequency; The relative pose relationships between each VR terminal are determined based on the positioning data packets obtained from the exchange.

3. The multi-VR terminal cooperative interaction method based on mutual positioning calculation of claim 1, wherein, The construction of a collaborative sensing network specifically includes: Obtain the relative pose relationships between various VR terminals; A collaborative perception network is constructed based on the relative pose relationships between various VR terminals.

4. The multi-VR terminal cooperative interaction method based on mutual positioning calculation of claim 1, wherein, Trajectory prediction is performed on the skeletal keypoints of the lost virtual character to obtain the motion prediction trajectory of the virtual character's skeletal keypoints, specifically including: The motion sequence of the virtual character whose observation was lost was obtained before the VR terminal occlusion interrupted, and the motion sequence included the position, velocity and acceleration of the skeletal key points; The motion sequence is used to predict the short-term trajectory of the skeletal key points of the virtual character in the early stage of occlusion, and the short-term predicted trajectory of the skeletal key points of the virtual character is obtained. The motion sequence is used to predict the long-term trajectory of the skeletal key points of the virtual character in the later stages of occlusion, and the long-term predicted trajectory of the skeletal key points of the virtual character is obtained. The short-term predicted trajectory and the long-term predicted trajectory are fused to obtain the motion prediction trajectory of the skeletal key points of the virtual character.

5. The multi-VR terminal cooperative interaction method based on mutual positioning calculation of claim 1, wherein, Based on the pose constraint boundary and the motion prediction trajectory, the virtual character whose observations have been lost is subjected to pose transition, and the pose transition parameters of the virtual character are obtained, specifically including: The pose transition of the virtual character with lost observation is optimized by using the pose constraint boundary and the motion prediction trajectory to obtain the position transition sequence of the root bone of the virtual character during the pose transition. The rotational transition sequence of the skeletal key points of the virtual character is determined based on the positional transition sequence; The pose transition parameters of the virtual character are determined by the position transition sequence and the rotation transition sequence.

6. The multi-VR terminal cooperative interaction method based on mutual positioning calculation of claim 1, wherein, Adjusting the spatial position of the virtual character through the posture transition parameters specifically includes: The virtual character's transitional display pose for each frame during the occlusion interruption is generated based on the posture transition parameters. When visual observation is restored, the virtual character's spatial position is updated using the virtual character's real-time pose and all transitional display poses.

7. The multi-VR terminal cooperative interaction method based on mutual positioning calculation of claim 1, wherein, The posture transition parameters represent a set of data that continuously drives the smooth movement of the virtual character during VR terminal occlusion interruptions.

8. A multi-VR terminal cooperative interaction system based on mutual positioning calculation, configured to perform the multi-VR terminal cooperative interaction method based on mutual positioning calculation according to any one of claims 1 to 7. The multi-VR terminal collaborative interaction system includes: The mutual positioning module is used by multiple VR terminals to locate each other by exchanging visual data and ranging data, thereby building a collaborative perception network; The credibility assessment module is used by each VR terminal as an observer to assess the pose credibility of the virtual character mapped by each VR terminal in the collaborative perception network, and obtain the observation credibility of the pose of each virtual character. The pose constraint module is used to broadcast the last observation information of the virtual character whose visual observation is lost to the cooperative perception network when all VR terminals detect the existence of a virtual character whose visual observation is lost, and to determine the pose constraint boundary of the virtual character whose observation is lost during the VR terminal occlusion interruption based on all the last observation information and all the observation confidence. The trajectory prediction module is used to predict the trajectory of the skeletal key points of the virtual character that have been lost in observation, and to obtain the motion prediction trajectory of the skeletal key points of the virtual character. The posture transition module is used to perform posture transition on the virtual character whose observation has been lost according to the posture constraint boundary and the motion prediction trajectory, to obtain the posture transition parameters of the virtual character, and then adjust the spatial position of the virtual character through the posture transition parameters.

Images