A dynamic reference-based virtual object pose determination method and system

Abstract

The present application relates to the field of mixed reality, and particularly relates to a virtual object posture determination method and system based on dynamic reference objects, and provides a virtual object posture determination method and system based on dynamic reference objects, which comprises: acquiring a video stream frame sequence photographed by a target user terminal, the video stream frame sequence comprising a plurality of time-continuous photographed images containing dynamic reference objects, determining a feature region in the photographed image with a confidence greater than a preset confidence threshold, determining a relative pose of the dynamic reference object according to a feature point in the feature region, estimating an optimal pose of the dynamic reference object based on the relative pose of the dynamic reference object, determining a shared dynamic reference pose of the dynamic reference object according to the optimal pose and a dynamic reference object pose detected by the online user terminal, and determining a pose of a virtual object based on the shared dynamic reference pose. Through the dynamic reference object as a space and posture reference, high-precision synchronization of the virtual object posture is realized in a multi-user scenario.

Description

Technical Field

[0001] This invention relates to the field of mixed reality technology, and in particular to a method and system for determining the pose of virtual objects based on dynamic reference objects. Background Technology

[0002] In the current field of mixed reality (MR), to enable collaborative operations of shared objects by multiple users in the same virtual scene, some technical solutions establish a unified spatial reference mechanism and state synchronization strategy to coordinate virtual perspectives and object interactions among multiple users. However, most current technologies rely on static visual anchor points as the basis for spatial alignment. Once a user moves out of the anchor point's field of view or the anchor point is occluded, the consistency of spatial coordinates is disrupted. This disruption may cause virtual objects to drift in position or rotate at inconsistent angles between different user terminals, thus affecting the continuity and accuracy of pose calculations. Summary of the Invention

[0003] To address the aforementioned problems in the prior art, this invention provides a method and system for determining the pose of a virtual object based on a dynamic reference frame. This method solves the problem that the virtual object's position drift or rotation angle is inconsistent between different user terminals, thus affecting the continuity and accuracy of pose calculation.

[0004] According to one aspect of the present invention, a method for determining the pose of a virtual object based on a dynamic reference frame is provided, comprising:

[0005] Acquire a sequence of video frames captured by a target user terminal; wherein the sequence of video frames includes multiple temporally consecutive images containing dynamic reference objects;

[0006] Identify the feature regions in the captured image whose confidence level is greater than a preset confidence threshold;

[0007] The relative pose of the dynamic reference object is determined based on the feature points in the feature region;

[0008] Based on the relative pose of the dynamic reference object, estimate the optimal pose of the dynamic reference object;

[0009] Based on the optimal pose and the pose of the dynamic reference object detected by the online user terminal, the shared dynamic reference pose of the dynamic reference object is determined.

[0010] The pose of the virtual object is determined based on the shared dynamic reference pose; wherein the virtual object and the dynamic reference are in the same coordinate system.

[0011] According to another aspect of the present invention, a virtual object pose determination system based on dynamic reference frames is provided, comprising:

[0012] A dynamic reference recognition module is used to acquire a video stream frame sequence captured by a target user terminal; wherein, the video stream frame sequence includes multiple temporally consecutive captured images containing dynamic reference objects; identify feature regions in the captured images with a confidence level greater than a preset confidence threshold; determine the relative pose of the dynamic reference object based on feature points in the feature regions; estimate the optimal pose of the dynamic reference object based on the relative pose of the dynamic reference object; and determine the shared dynamic reference pose of the dynamic reference object based on the optimal pose and the dynamic reference object pose detected by the online user terminal.

[0013] The pose fusion calculation module is used to determine the pose of the virtual object based on the shared dynamic reference pose; wherein the virtual object and the dynamic reference are in the same coordinate system.

[0014] The beneficial effects of this invention are reflected in the fact that it provides a method and system for determining the pose of a virtual object based on a dynamic reference object, comprising: acquiring a video stream frame sequence captured by a target user terminal; wherein the video stream frame sequence includes multiple temporally consecutive captured images containing a dynamic reference object; identifying feature regions in the captured images with a confidence level greater than a preset confidence threshold; determining the relative pose of the dynamic reference object based on feature points in the feature regions; estimating the optimal pose of the dynamic reference object based on its relative pose; determining a shared dynamic reference pose of the dynamic reference object based on the optimal pose and the dynamic reference object pose detected by the online user terminal; and determining the pose of the virtual object based on the shared dynamic reference pose; wherein the virtual object and the dynamic reference object are in the same coordinate system. Most existing MR collaborative systems rely on static anchor points (such as QR codes or fixed images) for spatial coordinate alignment. However, in multi-person mobile collaborative scenarios, static anchor points are easily occluded, offset, or removed from the field of view, resulting in coordinate system distortion and virtual object drift. This invention introduces a movable cursor, graphic, or active marker as a reference benchmark through a "dynamic reference recognition module." Combined with a multi-view PnP algorithm and a pose consistency fusion mechanism, it updates the global reference coordinate system in real time, making spatial alignment independent of fixed scene layouts and significantly improving adaptability to occlusion and changing environments. This scheme mathematically constructs a cross-user shared reference pose, ensuring that the terminal maintains spatial consistency with virtual objects even during viewpoint switching or movement. Therefore, by introducing dynamically identifiable reference objects as spatial and pose benchmarks, this invention constructs a unified and real-time updatable coordinate system, achieving high-precision synchronization and conflict-ordered control of virtual object poses in multi-user scenarios. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the virtual object pose determination method based on dynamic reference objects provided by the present invention.

[0016] Figure 2This is a flowchart illustrating a virtual object pose determination method based on a dynamic reference object, provided in another embodiment of the present invention.

[0017] Figure 3 This is a schematic diagram of the structure of the virtual object pose determination system based on dynamic reference provided by the present invention.

[0018] Figure 4 This is a schematic diagram of a virtual object pose determination system based on dynamic reference objects, provided in another embodiment of the present invention.

[0019] Figure 5 This is a structural diagram of an electronic device provided in an exemplary embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] Example 1: Figure 1 This is a flowchart illustrating the virtual object pose determination method based on dynamic reference objects provided by the present invention. Figure 2 This is a flowchart illustrating a virtual object pose determination method based on dynamic reference frames, provided in another embodiment of the present invention. Figures 1-2 As shown, the virtual object pose determination method based on dynamic reference frames includes:

[0022] S110: Acquire a video stream frame sequence captured by the target user terminal, wherein the video stream frame sequence includes multiple time-continuous captured images containing dynamic reference objects.

[0023] In this embodiment of the invention, since traditional anchor points (such as QR codes, static images, etc.) are easily occluded or lack dynamic adaptability, they cannot meet the requirements for continuous and high-precision updates of virtual object postures in a multi-person collaborative environment. Therefore, a dynamic visual reference object (hereinafter referred to as a dynamic reference object) is used as a posture reference benchmark, and a reference object detection algorithm based on multi-view projection is combined to achieve stable recognition of its spatial position and posture. The dynamic reference object can be a moving marker with a light source or a dynamic graphic identifier.

[0024] Specifically, the video stream frame sequence of the dynamic reference object is acquired in real time through the camera sensor of the target user terminal. The video stream frame sequence includes multiple temporally consecutive images containing the dynamic reference object.

[0025] S120: Identify the feature regions in the captured image whose confidence level is greater than the preset confidence threshold.

[0026] In this embodiment of the invention, in order to locate the position of the dynamic reference object in the captured image, the feature region in the captured image with a confidence level greater than a preset confidence threshold is determined as the region where the dynamic reference object is located.

[0027] S120 may include the following steps:

[0028] S121: Perform feature extraction on the captured image to obtain a set of candidate pixels, wherein the set of candidate pixels includes multiple candidate pixels.

[0029] In this embodiment of the invention, feature extraction is performed on each captured image in the video stream frame sequence. The dynamic reference object can be located using an image feature extraction method combining color, structural edges, and light spot distribution. Specifically, the image is converted to different color spaces (such as HSV, Lab, etc.), and color features are extracted using color histograms or color clustering (such as K-means). A color range matching the dynamic reference object is selected to generate a color mask. Edge detection algorithms (such as Canny or Sobel) are used to extract edge information in the image to obtain shape features. The edge information and color features are combined to further filter possible reference object regions. Light spot distribution information is used to identify regions with abrupt changes in brightness in the image, especially at feature points of the dynamic reference object. Hough transform can be used to detect specific shapes (such as circles or straight lines) and extract light spot features. The extracted color, structural edges, and light spot distribution features are fused to obtain a comprehensive feature. Each comprehensive feature is a candidate pixel, and a candidate pixel set is constructed using multiple pixels.

[0030] S122: Calculate the edge response value of each candidate pixel based on the brightness value of each candidate pixel.

[0031] In this embodiment of the invention, to enhance robustness, a weighted edge response function is introduced for region extraction under conditions of complex background or lighting fluctuations, defining candidate pixels in the image. The edge response is:

[0032]

[0033] in, Indicates time The image below shows candidate pixels. brightness value, , These represent the first-order gradients of the captured image in the horizontal and vertical directions, respectively. , These are directional weighting coefficients used to enhance regions with strong edges in a specific direction.

[0034] S123: Select target pixels with edge response values ​​greater than a preset response threshold from multiple candidate pixels.

[0035] In this embodiment of the invention, by calculating the edge response values ​​of candidate pixels, important information in the image can be effectively extracted. The candidate pixel set S is then filtered to meet the following criteria: The pixels are used as high-confidence regions, where For the response threshold, These are the pixel coordinates of the candidate pixels. Substitute to From which can be obtained Subsequently, the candidate pixels in the candidate pixel set S are matched with the predefined feature template of the reference object, and the candidate pixels with a matching degree greater than a preset matching degree threshold are selected as the target pixels.

[0036] S124: Determine the region constructed from multiple target pixels as a feature region with a confidence level greater than a preset confidence threshold.

[0037] S130: Determine the relative pose of the dynamic reference object based on the feature points in the feature region.

[0038] S140: Estimate the optimal pose of the dynamic reference object based on its relative pose.

[0039] S150: Determine the shared dynamic reference pose of the dynamic reference object based on the optimal pose and the dynamic reference object pose detected by the online user terminal.

[0040] In this embodiment of the invention, to improve the consistency and stability of recognizing the same dynamic reference object from multiple user perspectives, a multi-view cross-validation mechanism is further introduced. By introducing this mechanism, the dynamic reference poses of each user are shared through a P2P channel, forming a set of poses detected by all users. These dynamic reference poses are then fused using a weighted average method to generate a unified shared dynamic reference pose, thereby improving the system's localization consistency and stability in multi-user collaborative scenarios. Specifically, by dynamically adjusting the weights of each user terminal, considering signal quality and viewpoint occlusion, the pose fusion effect is optimized, ensuring that different user terminals can accurately identify the same dynamic reference object under different conditions. This improves the dynamic adaptability of spatial reference alignment and the localization consistency and stability in multi-user collaborative scenarios.

[0041] Specifically, the pose information of the reference object detected locally by each user will be broadcast through the P2P channel and a pose set will be established in the local cache. ,in This indicates the number of currently online user terminals; Index for users, ; Indicates the first The dynamic reference object rotation matrix calculated by each user terminal defines the spatial rotation relationship between the world coordinate system and the dynamic reference object coordinate system; Indicates the first The translation vector of the dynamic reference object calculated by each user terminal defines the position of the reference point of the dynamic reference object in the world coordinate system.

[0042] By taking a weighted average of the poses in the set, the shared dynamic reference pose is obtained:

[0043] ,

[0044]

[0045] in, To be assigned to the The weight of each user terminal is usually dynamically set based on signal quality or viewpoint obstruction. Describing the Frobenius norm, To share dynamic reference pose, Indicates the first The dynamic reference rotation matrix calculated by each user terminal defines the spatial rotation relationship between the world coordinate system and the dynamic reference coordinate system. `arg min` represents the variable itself that minimizes the objective function. `R` is the rotation matrix to be solved. Additionally, a specific value is found among all the rotation matrices `R` to be solved. This makes it consistent with the information provided by each user terminal. The weighted distance sum is minimized.

[0046] S160: Determine the pose of the virtual object based on the shared dynamic reference pose; wherein the virtual object and the dynamic reference are in the same coordinate system.

[0047] In this embodiment of the invention, in a multi-user collaborative mixed reality environment, different users acquire local spatial mapping data through their own devices (user terminals) and estimate the position and orientation of virtual objects from their respective perspectives by combining the poses of dynamic reference objects. Due to factors such as device heterogeneity, network latency, and observation conditions, the pose estimations of the same virtual object by different terminals often differ. Therefore, this application, guided by a dynamic reference coordinate system, performs unified mapping, weighted fusion, and rotation constraint fusion on multi-source pose data to output stable and synchronized consistent pose results.

[0048] Example 2: In one embodiment, S140 can be specifically implemented as follows: determining the three-dimensional coordinates corresponding to the target pixel of the dynamic reference object; obtaining the two-dimensional pixel coordinates corresponding to the target pixel; and estimating the optimal pose of the dynamic reference object based on the three-dimensional coordinates and the two-dimensional pixel coordinates corresponding to the target pixel of the dynamic reference object.

[0049] In this embodiment of the invention, the three-dimensional coordinates of the target pixel in the world coordinate system are utilized by a dynamic reference object. And its projected two-dimensional pixel coordinates in the corresponding captured image. Solve for the relative pose of a dynamic reference object. ,in Represents the rotation matrix. Let be the translation vector. According to the pinhole camera model, the following projection relationship exists:

[0050]

[0051] in, As a scaling factor, This is the intrinsic parameter matrix of the camera. Represents the relative pose of a dynamic reference object. .

[0052] Based on the relative pose of the dynamic reference object The optimal pose of the dynamic reference object is estimated. Specifically, the optimal pose of the dynamic reference object is estimated by minimizing the following reprojection error function:

[0053] ;

[0054] in, This function represents the projection of three-dimensional coordinates onto the image plane. Denotes the Euclidean norm. The number of feature points used for matching on the reference object. The optimal pose for a dynamic reference object. The coordinates of the i-th feature point of the dynamic reference object in the current image frame are the two-dimensional pixel coordinates.

[0055] Example 3: In one embodiment, S160 can be specifically implemented as follows: obtaining the estimated pose of the online user terminal relative to the shared dynamic reference pose; determining the position information of the shared reference object corresponding to the shared dynamic reference pose in the world coordinate system; and estimating the pose of the virtual object based on the estimated pose and the position information.

[0056] In this embodiment of the invention, let the first... The locally estimated pose of each user terminal relative to the shared dynamic reference is: ,in Represents a three-dimensional rotation matrix. Represents the translation vector. If the position of the shared reference object in the world coordinate system is... Then, the preliminary estimated pose of the virtual object in the global space calculated by the user terminal is:

[0057]

[0058] ;

[0059] in, This is the estimated pose of the online user terminal relative to the shared dynamic reference pose. The estimated pose is the local relative pose estimate of the nth user terminal relative to the dynamic reference object, which is the result calculated independently by each user terminal and lies between the user's own coordinate system and the reference object's coordinate system. , [This represents the initial estimated pose of the virtual object in global space.] This is the translation vector of the virtual object in global space. This is the three-dimensional rotation matrix of the virtual object in global space.

[0060] Example 4: In one embodiment, S160 can be specifically implemented as follows: estimating the pose of the virtual object in the global space based on the estimated pose and position information; wherein, the pose in the global space includes a rotation matrix and a translation vector in the global space; converting the rotation matrix in the global space into a rotation quaternion; constructing a matrix to be processed based on the rotation quaternion; obtaining a fusion result based on the matrix to be processed; and determining the pose of the virtual object based on the fusion result and the translation vector in the global space.

[0061] In this embodiment of the invention, when fusing the poses of virtual objects from multiple user terminals in global space, to avoid local errors affecting overall accuracy, the system introduces a rotation fusion method based on quaternion weighted averaging. This method converts the rotation matrix in global space into rotation quaternions. Let the first The unit quaternion reported by each user terminal is Its corresponding weight is The system expects to obtain an optimal fusion quaternion. The goal is to make it, in a weighted sense, closest to the pose of the virtual object in global space estimated by all user terminals.

[0062] ;

[0063] in, for transpose, Let C be a rotation quaternion and C be the matrix to be processed.

[0064] To solve this problem, a weighted covariance matrix is ​​constructed, which is the matrix to be processed, C:

[0065] ;

[0066] in, for transpose, Let C be a rotation quaternion and C be the matrix to be processed.

[0067] Example 5: In one example, S160 can be specifically implemented as follows: decompose the matrix to be processed to obtain the decomposition result; determine the eigenvector corresponding to the largest eigenvalue in the decomposition result as the fusion result.

[0068] In this embodiment of the invention, the matrix to be processed... The eigenvalue decomposition is performed, and the eigenvector corresponding to the largest eigenvalue is the fusion result. This method possesses excellent rotational smoothing properties and the ability to maintain consistency across multiple data sources, making it suitable for pose fusion in asynchronous collaborative scenarios.

[0069] For the translation vector part, a weighted arithmetic mean is used to fuse the global translation estimates from each terminal. :

[0070] ;

[0071] Where the weight is , Let be the translation vector in the global space.

[0072] In this embodiment of the invention, [q, t] will be used as the standard state of the virtual object, broadcast to all user terminals, and serve as the basis for subsequent control coordination and state synchronization. This effectively solves the problems of posture deviation, drift and asynchrony in multi-user collaborative scenarios, and significantly improves the consistency, stability and real-time performance of the virtual object in multi-terminal presentation.

[0073] Example 6: In one embodiment, after step S160, the virtual object pose determination method based on dynamic reference objects can be specifically implemented as follows: acquiring operation data of the online user terminal; wherein, the operation data includes the rotation quaternion corresponding to the online user terminal during operation and the three-dimensional translation vector corresponding to the online user terminal during operation; optimizing the pose of the virtual object based on the rotation quaternion corresponding to the operation and the three-dimensional translation vector corresponding to the operation.

[0074] In this embodiment of the invention, to coordinate the pose update information of virtual objects from multiple user terminals and ensure that all virtual objects observed by users in a multi-terminal mixed reality environment are always in a unified and continuous state, object operation events uploaded by each user terminal are received in real time. The state stream is then sorted, compared, and synchronously broadcast based on multi-dimensional parameters such as timestamps, frame sequences, and network latency. This ensures interaction consistency and smooth action response in high-frequency interactive scenarios.

[0075] To achieve cross-terminal synchronization of virtual object states, the data packets uploaded by user terminals are first encapsulated in a time sequence. Let the data packets of the i-th user terminal at time... The generated operation data is represented as ,in This represents the rotation quaternion corresponding to the operation of the online user terminal. This refers to the three-dimensional translation vector corresponding to the online user terminal during operation. This refers to the type of operation event (e.g., grab, move, rotate, etc.). To avoid state instability caused by network jitter, each data packet includes a local timestamp. And time reference calibration is performed through an end-to-end time synchronization mechanism.

[0076] Once the system receives the data packets from all online users, the core issue that needs to be addressed is how to globally synchronize the timeline. The module restores the most likely unified state of the virtual objects. To this end, it introduces a timestamp-aligned fusion scheduling function to perform minimum latency alignment on all data. Let the target synchronization time point be... Then the system needs to select a set of attitude-state combinations. Satisfy the following minimum cost criterion:

[0077] ;

[0078] in, and Here, represents the synchronization weight parameters in the rotation and translation directions, respectively; q represents the fusion result; and t represents the translation vector in the global space. Local timestamp. Dynamically adjusted based on current user network latency, operation frequency, historical error statistics, etc. For the i-th user at local time The rotational attitude (quaternion). For the i-th user at local time The corresponding three-dimensional translation vector during the operation. The attitude solution is obtained by solving this optimization problem. This indicates the pose that best maintains consistency across all user terminal observations during the current broadcast cycle. The system then synchronously broadcasts this standard pose value, and each terminal updates its local virtual object state accordingly. This achieves consistency in the position and orientation of virtual objects across devices and networks, avoiding jitter or state conflicts caused by network latency, observation differences, and other factors. It significantly improves the smoothness of interaction and the consistency of experience in multi-user collaborative scenarios.

[0079] To further improve synchronous robustness and adapt to asynchronous network environments, this module introduces a buffering mechanism based on a sliding time window. Let the system broadcast frame rate be... Then each During the period, This indicates the time interval between each frame; the system maintains a rolling time window. Only data within that time period will be used for fusion calculations. This represents the timestamp indicating the time when the current broadcast cycle best maintains consistent posture across all user terminal observations, i.e., the synchronization target time point. Window length It can adaptively adjust according to network fluctuation levels, thereby achieving an optimal balance between latency and synchronization accuracy. All data that does not arrive within the time window will be delayed until the next cycle to avoid global state jumps and object motion jitter.

[0080] Furthermore, in cross-regional collaboration or 5G edge deployment scenarios, the system supports the use of edge synchronization proxy nodes for localized data preprocessing and state fusion. Within each edge node, after executing the aforementioned fusion and scheduling algorithms, a unified state value is pushed to each user terminal with low latency via a relay channel, significantly shortening the synchronization response link and enhancing the system's scalability and deployment flexibility.

[0081] Example 7: In one embodiment, after step S160, the virtual object pose determination method based on dynamic reference objects can be specifically implemented as follows: if multiple online user terminals initiate control requests for virtual objects, then obtain the position of each online user terminal relative to the virtual object and the current position of the virtual object; obtain the credibility score of each online user terminal; obtain the submission delay corresponding to each online user terminal initiating control requests for virtual objects; determine the control priority of each online user terminal based on the position of each online user terminal relative to the virtual object, the current position of the virtual object, the credibility score of each online user terminal, and the submission delay; determine the target online user terminal with the highest control priority to control the virtual object.

[0082] In this embodiment of the invention, the control conflict problem arising when multiple users simultaneously attempt to control the same virtual object is resolved, ensuring the uniqueness of object control rights, the real-time nature of operations, and the continuity of state at each moment. Based on the standardized attitude and state flow provided by the multi-user synchronization module, and combined with real-time operation intentions from different terminals, this module uses a dynamic conflict judgment mechanism and priority control strategy to schedule and adjudicate user control requests in real time, thereby achieving control consistency assurance in high-concurrency interactions.

[0083] In a multi-user collaborative mixed reality scenario, let the set of online users be... , This represents the nth online user. When multiple users simultaneously initiate control requests for this object, the system needs to judge and handle the conflict based on multiple factors such as user attributes, spatial status, and operation sequence. Each online user The submitted control request can be represented as an operation tuple. ,in For operation types (such as move, rotate, pick). This indicates the position of each online user terminal relative to the virtual object. Indicates the local timestamp of the operation.

[0084] To ensure reasonable allocation of control, this module introduces a dynamic evaluation mechanism based on a priority scoring function. The function defining the control priority of each user terminal for the current virtual object is as follows:

[0085]

[0086] in, This represents the Euclidean distance between the user terminal and the virtual object. The current position of the virtual object, that is , This indicates the credibility score of the current operation of the online user terminal (such as historical success rate, task role level, etc.). Indicates the commit latency of the operation, where This represents the global target synchronization time point set by the system in the multi-user synchronization module. Indicates the first Each user terminal operates its data Additional local timestamp; For the corresponding weighting coefficients, It is a tiny constant used to avoid division by zero errors.

[0087] The system is based on all The values ​​(control priority) are compared in real time, and the user corresponding to the largest value is selected. Granting control, that is:

[0088]

[0089] Among them, symbols This represents the set of users who are currently qualified to operate on objects and participate in the control competition. Any element in the set... Corresponding to the Each specific user terminal submits an operation request to the system once in the current logical frame. For each user... The system will calculate the control priority score based on factors such as operation time, age, spatial proximity, and role permission level. This score, a real number, measures the relative rationality and priority of a user acquiring control in the current frame. The system compares the control priority scores of all users in set U and selects the index of the user with the highest score. The corresponding users The system grants sole control over the current virtual object and broadcasts a lock notification to other terminals, thereby ensuring consistency and stability of object control during multi-user collaboration.

[0090] Once control is granted, other user terminals will immediately receive a notification that the object state has been locked and will perform a smooth inter-frame transition display based on the object's standard pose provided by the pose fusion calculation module. Operations performed by the user granted control will be broadcast with high priority throughout the control session and rapidly distributed via the multi-user synchronization module to maintain synchronized updates of the virtual object state on other terminals.

[0091] Example 8: In one embodiment, the virtual object posture determination method based on dynamic reference objects can be specifically implemented as follows: determining a minimum control hold time window; if other online user terminals issue control requests to the virtual object within the time window, then determining that the target online user terminal continues to control the virtual object; wherein, the control priority of other online user terminals is higher than the control priority of the target online user terminal.

[0092] To avoid inconsistent state issues caused by frequent switching of control, the system sets a minimum control hold time window. Within this window, even if other users have a higher priority score than the currently controlling user, a switch of control will not be triggered. Control will only be restored to the set of candidate users after the currently controlling user has been inactive for an extended period, experienced a network outage, or manually relinquished control.

[0093] If multiple objects are in a conflicting state simultaneously, the system will construct a priority scheduling graph for all conflicting objects, using objects as nodes and conflict requests as edges, and perform sequential scheduling or centralized arbitration to ensure that operation paths do not intersect or block. In complex collaborative scenarios, this module can also be linked with the task management module to set upper limit boundaries for control requests based on task allocation permissions, further improving the predictability and controllability of control scheduling.

[0094] Figure 3 This is a schematic diagram of the structure of the virtual object pose determination system based on dynamic reference frames provided by the present invention. Figure 3 As shown, the virtual object pose determination system based on dynamic reference objects includes: a dynamic reference recognition module, used to acquire a video stream frame sequence captured by a target user terminal; wherein, the video stream frame sequence includes multiple temporally consecutive captured images containing dynamic reference objects; identifying feature regions in the captured images with a confidence level greater than a preset confidence threshold; determining the relative pose of the dynamic reference object based on feature points in the feature regions; estimating the optimal pose of the dynamic reference object based on its relative pose; determining the shared dynamic reference pose of the dynamic reference object based on the optimal pose and the dynamic reference object pose detected by the online user terminal; and a pose fusion calculation module, used to determine the pose of the virtual object based on the shared dynamic reference pose; wherein, the virtual object and the dynamic reference object are in the same coordinate system.

[0095] Figure 4 This is a schematic diagram of a virtual object pose determination system based on a dynamic reference frame, provided in another embodiment of the present invention. Figure 4 As shown, the dynamic reference recognition module can be specifically configured as follows: performing feature extraction on the captured image to obtain a set of candidate pixels; wherein the set of candidate pixels includes multiple candidate pixels; calculating the edge response value of each candidate pixel based on the brightness value of each candidate pixel; selecting target pixels with edge response values ​​greater than a preset response threshold from among the multiple candidate pixels; and determining the region constructed by the multiple target pixels as a feature region with a confidence level greater than a preset confidence threshold.

[0096] In one embodiment, the dynamic reference recognition module may be specifically configured to: determine the three-dimensional coordinates corresponding to the target pixel of the dynamic reference object; obtain the two-dimensional pixel coordinates corresponding to the target pixel; and estimate the optimal pose of the dynamic reference object based on the three-dimensional coordinates and the two-dimensional pixel coordinates corresponding to the target pixel of the dynamic reference object.

[0097] In one embodiment, the pose fusion calculation module may be specifically configured to: obtain the estimated pose of the online user terminal relative to the shared dynamic reference pose; determine the position information of the shared reference object corresponding to the shared dynamic reference pose in the world coordinate system; and estimate the pose of the virtual object based on the estimated pose and the position information.

[0098] In one embodiment, the pose fusion calculation module can be specifically configured to: estimate the pose of the virtual object in the global space based on the estimated pose and position information; wherein the pose in the global space includes a rotation matrix and a translation vector in the global space; convert the rotation matrix in the global space into a rotation quaternion; construct a matrix to be processed based on the rotation quaternion; obtain the fusion result based on the matrix to be processed; and determine the pose of the virtual object based on the fusion result and the translation vector in the global space.

[0099] In one embodiment, the attitude fusion calculation module can be specifically configured to: decompose the matrix to be processed to obtain the decomposition result; and determine the eigenvector corresponding to the largest eigenvalue in the decomposition result as the fusion result.

[0100] In one embodiment, after the pose fusion calculation module, a multi-user synchronization module is used to acquire operation data of online user terminals; wherein, the operation data includes the rotation quaternion corresponding to the online user terminal during operation and the three-dimensional translation vector corresponding to the online user terminal during operation; and the pose of the virtual object is optimized based on the rotation quaternion and the three-dimensional translation vector corresponding to the operation.

[0101] In one embodiment, after the multi-user synchronization module, the object control coordination module is configured to, if multiple online user terminals initiate control requests for a virtual object, obtain the position of each online user terminal relative to the virtual object and the current position of the virtual object; obtain the credibility score of each online user terminal; obtain the submission delay corresponding to the control request initiated by each online user terminal to the virtual object; determine the control priority of each online user terminal based on the position of each online user terminal relative to the virtual object, the current position of the virtual object, the credibility score of each online user terminal, and the submission delay; and determine the target online user terminal with the highest control priority to control the virtual object.

[0102] In one embodiment, the delay compensation module may be specifically configured to: determine a minimum control hold time window; if other online user terminals issue control requests to the virtual object within the time window, then determine that the target online user terminal continues to control the virtual object; wherein, the control priority of other online user terminals is higher than the control priority of the target online user terminal.

[0103] Figure 5 A block diagram of an electronic device according to an embodiment of this application is illustrated.

[0104] like Figure 5 As shown, the electronic device includes one or more processors and memory.

[0105] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0106] The memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and a processor may execute the program instructions to implement the virtual object pose determination method based on dynamic reference objects of the various embodiments of this application described above, and / or other desired functions. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0107] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0108] When the electronic device is a standalone device, the input device can be a communication network connector for receiving the acquired input signals from the first device and the second device.

[0109] In addition, the input device may also include, for example, a keyboard, a mouse, etc.

[0110] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0111] Of course, for the sake of simplicity, Figure 5 Only some of the components of the electronic device relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device may include any other suitable components depending on the specific application.

[0112] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0113] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0114] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for determining the pose of a virtual object based on dynamic reference frames, characterized in that, include: Acquire a sequence of video frames captured by a target user terminal; wherein the sequence of video frames includes multiple temporally consecutive images containing dynamic reference objects; Identify the feature regions in the captured image whose confidence level is greater than a preset confidence threshold; The relative pose of the dynamic reference object is determined based on the feature points in the feature region; Based on the relative pose of the dynamic reference object, estimate the optimal pose of the dynamic reference object; Based on the optimal pose and the pose of the dynamic reference object detected by the online user terminal, a shared dynamic reference pose of the dynamic reference object is determined; wherein, determining the shared dynamic reference pose of the dynamic reference object includes: Establish a pose set in the local cache; where the formula for the pose set is: ,in This indicates the number of currently online user terminals. Index for users, , Indicates the first The dynamic reference rotation matrix calculated by each user terminal Indicates the first The dynamic reference object translation vector calculated by each user terminal; A weighted average of the poses in the set is used to obtain the shared dynamic reference pose; wherein, the calculation formula for the shared dynamic reference pose is: ； ,in, To be assigned to the The weight of each user terminal Describing the Frobenius norm, To share dynamic reference pose, Indicates the first The dynamic reference rotation matrix calculated by each user terminal, arg min represents the variable itself that makes the objective function reach its minimum value, and R is the rotation matrix to be solved; Based on the shared dynamic reference pose, the pose of the virtual object is determined; wherein, the virtual object and the dynamic reference are in the same coordinate system; The step of determining the feature regions in the captured image with a confidence level greater than a preset confidence threshold includes: Feature extraction is performed on the captured image to obtain a candidate pixel set; wherein, the candidate pixel set includes multiple candidate pixels; Based on the brightness value of each candidate pixel, calculate the edge response value of each candidate pixel; Target pixels with edge response values ​​greater than a preset response threshold are selected from multiple candidate pixels; The region constructed from multiple target pixels is identified as a feature region with a confidence level greater than a preset confidence threshold.

2. The virtual object pose determination method based on dynamic reference objects according to claim 1, characterized in that, The step of estimating the optimal pose of the dynamic reference object based on its relative pose includes: Determine the three-dimensional coordinates corresponding to the target pixel of the dynamic reference object; Obtain the two-dimensional pixel coordinates corresponding to the target pixel; The optimal pose of the dynamic reference object is estimated based on the three-dimensional coordinates of the target pixel and the two-dimensional pixel coordinates of the target pixel.

3. The virtual object pose determination method based on dynamic reference objects according to claim 1, characterized in that, Determining the pose of the virtual object based on the shared dynamic reference pose includes: Obtain the estimated pose of the online user terminal relative to the shared dynamic reference pose; Determine the position information of the shared reference object corresponding to the shared dynamic reference pose in the world coordinate system; The pose of the virtual object is estimated based on the estimated pose and the position information.

4. The virtual object pose determination method based on dynamic reference objects according to claim 3, characterized in that, The step of estimating the pose of the virtual object based on the estimated pose and the position information includes: Based on the estimated pose and the position information, the pose of the virtual object in the global space is estimated; wherein, the pose in the global space includes a rotation matrix and a translation vector in the global space; Convert rotation matrices in global space into rotation quaternions; Construct the matrix to be processed based on the rotation quaternion; Based on the matrix to be processed, obtain the fusion result; The pose of the virtual object is determined based on the fusion result and the translation vector in the global space.

5. The virtual object pose determination method based on dynamic reference objects according to claim 4, characterized in that, The step of obtaining the fusion result based on the matrix to be processed includes: The matrix to be processed is decomposed to obtain the decomposition result; The eigenvector corresponding to the largest eigenvalue in the decomposition result is determined as the fusion result.

6. The virtual object pose determination method based on dynamic reference objects according to claim 4, characterized in that, After estimating the pose of the virtual object based on the estimated pose and the position information, the method further includes: Acquire operation data of the online user terminal; wherein, the operation data includes the rotation quaternion and the three-dimensional translation vector corresponding to the operation of the online user terminal; The pose of the virtual object is optimized based on the rotation quaternion and the three-dimensional translation vector corresponding to the operation.

7. The virtual object pose determination method based on dynamic reference objects according to claim 6, characterized in that, After optimizing the pose of the virtual object based on the rotation quaternion and the three-dimensional translation vector corresponding to the operation, the method further includes: If multiple online user terminals initiate control requests for the virtual object, then the position of each online user terminal relative to the virtual object and the current position of the virtual object are obtained; Obtain a credibility score for each online user terminal; Obtain the submission delay corresponding to each online user terminal's control request for the virtual object; The control priority of each online user terminal is determined based on the position of each online user terminal relative to the virtual object, the current position of the virtual object, the credibility score of each online user terminal, and the submission delay. The target online user terminal with the highest control priority is determined to control the virtual object.

8. The virtual object pose determination method based on dynamic reference objects according to claim 7, characterized in that, After determining that the target online user terminal with the highest control priority controls the virtual object, the method further includes: Determine the minimum control hold time window; If other online user terminals issue control requests to the virtual object within the time window, it is determined that the target online user terminal will continue to control the virtual object; wherein, the control priority of the other online user terminals is higher than the control priority of the target online user terminal.

9. A virtual object pose determination system based on dynamic reference frames, characterized in that, include: A dynamic reference recognition module is used to acquire a video stream frame sequence captured by a target user terminal; wherein the video stream frame sequence includes multiple temporally consecutive captured images containing dynamic reference objects; identify feature regions in the captured images with a confidence level greater than a preset confidence threshold; determine the relative pose of the dynamic reference object based on feature points in the feature regions; estimate the optimal pose of the dynamic reference object based on the relative pose of the dynamic reference object; and determine the shared dynamic reference pose of the dynamic reference object based on the optimal pose and the dynamic reference object pose detected by the online user terminal; wherein determining the shared dynamic reference pose of the dynamic reference object includes: Establish a pose set in the local cache; where the formula for the pose set is: ,in This indicates the number of currently online user terminals. Index for users, , Indicates the first The dynamic reference rotation matrix calculated by each user terminal Indicates the first The dynamic reference object translation vector calculated by each user terminal; A weighted average of the poses in the set is used to obtain the shared dynamic reference pose; wherein, the calculation formula for the shared dynamic reference pose is: ； ,in, To be assigned to the The weight of each user terminal Describing the Frobenius norm, To share dynamic reference pose, Indicates the first The dynamic reference rotation matrix calculated by each user terminal, arg min represents the variable itself that makes the objective function reach its minimum value, and R is the rotation matrix to be solved; The pose fusion calculation module is used to determine the pose of a virtual object based on the shared dynamic reference pose; wherein the virtual object and the dynamic reference are in the same coordinate system; wherein determining the feature region in the captured image with a confidence level greater than a preset confidence threshold includes: extracting features from the captured image to obtain a set of candidate pixels; wherein the set of candidate pixels includes multiple candidate pixels; calculating the edge response value of each candidate pixel based on the brightness value of each candidate pixel; selecting target pixels with edge response values ​​greater than a preset response threshold from the multiple candidate pixels; and determining the region constructed by the multiple target pixels as the feature region with a confidence level greater than the preset confidence threshold.

Images