A multi-user mobile augmented reality system and method based on edge assistance

Abstract

This invention discloses an edge-assisted multi-user mobile augmented reality system and method. The system is designed based on a network environment consisting of a placer, a server, and followers. It includes a placer adaptive unloading module on the placer's end, an edge collaborative optimization module on the server's end, and a follower adaptive unloading module, an AR dynamic update module, and a relative pose estimation module on the follower's end. The system sequentially performs placer adaptive unloading, follower adaptive unloading, server edge collaborative optimization, and AR dynamic update processing, continuously optimizing the AR pose of the follower and obtaining real-time and accurate AR pose dynamic update results, which are then input into the follower's mobile device. This invention constructs an environment-aware multi-user MAR system through edge-server collaborative optimization, achieving continuous and highly accurate positioning and recognition in a multi-user mobile augmented reality system.

Description

Technical Field

[0001] This invention relates to the field of augmented reality technology, and in particular to a multi-user mobile augmented reality system. Background Technology

[0002] Augmented Reality (AR), as a bridge to the metaverse, holds significant importance at the national, technological, and social development levels. AR is a technology that integrates virtual information with the real world, allowing users to overlay virtual objects onto the camera's field of view to enhance the real world. With the widespread adoption of mobile devices, improved hardware performance, and advancements in communication network technology, research on multi-user AR based on mobile devices is receiving increasing attention. Compared to single-user Mobile Augmented Reality (MAR), multi-user MAR allows multiple users to interact, cooperate, or compete within a MAR system. Multi-user MAR interaction holds promise for providing immersive experiences in entertainment, education, and healthcare.

[0003] There is already some research on Markup Analysis (MAR). For example, in industry, Google has launched ARCore, which is based on edge cloud assistance, while Apple has launched ARKit, which is based on peer-to-peer interaction. In academia, it is roughly divided into two categories: MAR based on computer vision and MAR based on geolocation information. In recent years, with the rise of artificial intelligence, researchers have been working to integrate artificial intelligence algorithms into MAR systems.

[0004] Currently, industrial multi-user MAR systems focus more on application development. Because they are closed-source, little is known about the underlying platform's communication mechanisms. In academia, multi-user MAR systems based solely on object detection lack precise positioning capabilities, leading to inaccurate or delayed updates that disrupt the seamless integration of MAR with the real world, thus degrading the user experience. Point-to-point architecture solutions, due to joint computation on mobile devices, experience longer coordinate system alignment times, resulting in even longer overall end-to-end latency. Furthermore, the high computational complexity and energy consumption of artificial intelligence algorithms hinder their application in MAR systems. Summary of the Invention

[0005] Based on the aforementioned existing technologies, this invention proposes an edge-assisted multi-user mobile augmented reality system and method. By co-optimizing edge servers, an environment-aware multi-user MAR system is constructed, allowing multiple users to achieve accurate and real-time interaction in the same physical environment.

[0006] This invention is achieved using the following technical solution:

[0007] An edge-assisted multi-user mobile augmented reality system, based on a network environment consisting of placers, servers, and followers, includes a placer adaptive offloading module located at the placer end, an edge collaborative optimization module located at the server end, and a follower adaptive offloading module, an AR dynamic update module, and a relative pose estimation module located at the follower end; wherein:

[0008] The placer adaptive unloading module selects the placer's keyframe, associated point cloud, and the placer's current AR pose, and unloads it to the server; and connects to the relative pose estimation module to solve a rough AR pose through relative pose estimation and send it to the follower.

[0009] The edge collaboration optimization module obtains more robust feature matching point pairs between the placer and the follower through a strict loose matching algorithm, thereby obtaining a more accurate relative positional relationship between the two mobile devices, and thus continuously optimizing the AR pose of the follower.

[0010] The follower adaptive unloading module selects the follower's keyframes, associated point clouds, and the follower's current pose, and unloads them to the server.

[0011] The AR dynamic update module handles the AR pose dynamic update of the follower end to adapt to changes in time and space.

[0012] An edge-assisted multi-user mobile augmented reality method, the method comprising the following steps;

[0013] Step 1: Perform adaptive unloading processing for the placer. On the one hand, select the keyframes of the placer, the associated point cloud, and the pose of the placer at this time to form a packet file on the placer side and unload it to the server. On the other hand, perform system initialization and use relative pose estimation to solve a rough AR pose and send it directly to the mobile device on the follower side.

[0014] In addition, follower adaptive setup and offloading processing is performed, that is, the keyframe of the current follower, the associated point cloud and the current pose of the follower are selected to form a packet file on the follower side and offloaded to the server.

[0015] Step 2: Perform edge collaborative optimization processing. Obtain feature matching point pairs through a strict loose matching algorithm. Calculate the relative positional relationship R and t between the placer and the follower using the feature matching point pairs, where R represents the rotation matrix and t represents the translation vector. This allows for understanding the contextual information in the environment and obtaining accurate scene recognition results.

[0016] Step 3: Perform AR dynamic update processing, which means reusing the pose of the keyframe uploaded by the follower to the server and the current pose, and dynamically updating the rough AR pose in Step 1 to adapt the edge collaborative optimization results to changes in time and space, continuously optimizing the AR pose of the follower, and obtaining real-time and accurate AR pose dynamic update results to the mobile device of the follower.

[0017] Compared with existing technologies, this invention achieves continuous and highly accurate high-precision positioning in multi-user mobile augmented reality systems, and achieves the following beneficial technical effects:

[0018] 1) An edge collaborative optimization strategy is proposed to continuously optimize the MAR pose of multiple users and ensure its accuracy;

[0019] 2) An AR dynamic update strategy is proposed, which is based on the high-precision positioning provided by Simultaneous Localization and Mapping (SLAM) technology to ensure the effectiveness of multi-user MAR as time and space change;

[0020] 3) An adaptive offloading strategy is proposed to adapt to different network conditions and ensure the real-time performance of multi-user MAR. Attached Figure Description

[0021] Figure 1 This is a structural diagram of the edge-assisted multi-user mobile augmented reality system of the present invention;

[0022] Figure 2 Flowchart for the edge collaboration optimization module;

[0023] Figure 3 Flowchart of the strict loose matching algorithm;

[0024] Figure 4 Flowchart for the AR dynamic update module;

[0025] Figure 5 This is a flowchart for the adaptive unloading module. Detailed Implementation

[0026] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0027] like Figure 1The diagram shows a block diagram of the edge-assisted multi-user mobile augmented reality system of the present invention. The system is designed based on a network environment consisting of a placer 1, a server 2, and a follower 3. It includes a placer adaptive offloading module 100 located at the placer 1 end, an edge collaborative optimization module 200 located at the server 2 end, and a follower adaptive offloading module 300, an AR dynamic update module 400, and a relative pose estimation module 500 located at the follower 3 end. The mobile device 600 with AR placement is the placer 1 end, and the mobile device 700 with AR rendering recovery is the follower 2 end.

[0028] The adaptive unloading module 100, located at the placer 1, selects keyframes, associated point clouds, and the placer's current pose from the input of the mobile device 600 placing the AR, forming a packet file for unloading to server 2. Simultaneously, it initializes the system and directly sends data to the follower. The relative pose estimation module allows the follower to calculate a coarse AR pose using relative pose estimation. Two keyframe selection principles are used: Principle 1: Select the frame at the moment the mobile device receives the dropped message from the edge server as the keyframe; Principle 2: Select the frame when the current frame is significantly different from the most recently unloaded frame. Principle 1 avoids network congestion on the uplink. Principle 2 saves communication and computation costs. This allows both the placer and the follower to better adapt to different environments, enabling the edge server to perform its optimal optimization function.

[0029] The edge collaborative optimization module 200, located on server 2, continuously optimizes the AR pose of the follower using edge collaborative perception. It also obtains more robust feature matching point pairs through a strict loose matching algorithm. The feature matching point pairs are used to calculate a more accurate relative positional relationship R and t between the placer and the follower, where R represents the rotation matrix and t represents the translation vector. This module understands the contextual information in the environment through target detection and other means, thereby improving the accuracy of scene recognition.

[0030] The follower adaptive setup and unloading module 300, set on the follower 3 end, selects the follower's keyframes, associated point clouds, and the follower's current pose by rendering the input of the AR mobile device 700, forming a package file, and unloading it to the server 2.

[0031] While one of the simplest ways to share AR information between placers and followers is to send all of the placer's SLAM data to the followers, this results in significant communication latency, making multi-user MAR very ineffective. The MAR device also continuously offloads camera frames to the server via a wireless link, requiring substantial bandwidth. Therefore, an adaptive offloading strategy is needed to reduce redundant data transmission and ensure real-time performance.

[0032] The AR dynamic update module 400, located on the follower 3, handles the dynamic update of the follower's AR pose to adapt to changes in time and space. Since the mobile device can move freely, the relative position information R and t downloaded from the server may no longer be applicable to the current state. This module obtains the final accurate AR pose on the follower. Specifically, based on the SLAM system running on the mobile device, it reuses the pose of the keyframes uploaded by the follower to the server and the current pose, and then solves for the accurate and real-time AR pose of the follower through a series of coordinate system transformations.

[0033] Specifically, the edge collaborative optimization module 200 includes keyframe preprocessing to determine whether two frames are in the same environment: YOLOv5x object detection algorithm is used to perform object detection on both keyframes, outputting object type, detection confidence, and their two-dimensional bounding boxes, thus obtaining the environmental information contained in the keyframes; the object type of the follower is compared with the object type of the placer; if the matching rate is greater than the established threshold T... match If both keyframes are present, it indicates that the two unloading keyframes are in the same environment, and the system can proceed to the next stage. Otherwise, these two frames are discarded, and this information is fed back to the mobile device. This step involves the edge server performing keyframe preprocessing after receiving the unloading keyframes from the placer and follower.

[0034] Specifically, the edge collaborative optimization module 200 includes feature point preprocessing: due to the influence of moving objects in the scene, some abnormal feature points will appear in the point cloud. This will lead to mismatches in feature matching, thus affecting the accuracy of R and t. Abnormal feature points are removed to obtain more robust feature points in the keyframe. The preprocessing process is as follows: the input is the relevant point cloud information and the poses of the placer and follower keyframes. The output is the filtered relatively static feature points. First, the point cloud is preprocessed; by using the pose of the unloading keyframe as an extrinsic parameter and the intrinsic parameters of the mobile device camera itself, the pixel coordinates of the point cloud on the keyframe are calculated; then, dynamic feature point culling is performed. The above process relies on object matching in the previous object detection, inferring changes by comparing the relative pixel positions of the matching boxes in the keyframe. The key principle of this process is that if an object moves in the frame, its relative positional relationship with other objects in the frame will change. In this way, feature points on highly dynamic objects (such as people) can be removed.

[0035] like Figure 3 The diagram shown is a flowchart of the strict loose matching algorithm. The specific process of the strict loose matching algorithm in the edge collaborative optimization module is as follows:

[0036] Input feature points on the keyframe;

[0037] Determine whether a feature point is a feature point on a relatively static object. If yes, perform loose matching; otherwise, perform strict matching.

[0038] Output more and more robust feature matching point pairs.

[0039] In summary, by performing strict matching of feature points in the background and loose matching of feature points on relatively static objects, more robust feature matching point pairs can be obtained between two keyframes.

[0040] In other words, the overall idea of ​​this algorithm is to loosely match feature points on relatively static objects while strictly matching feature points in the background environment. The key behind this algorithm is that feature points on static objects are more stable, allowing for loose matching, while feature points in the environment are more susceptible to noise and therefore require strict matching. The system can then use 3D-2D PNP or 3D-3D ICP methods to obtain the relative positional relationship between two keyframes. Subsequently, accurate R and t values ​​are obtained through nonlinear optimization (Bundle Adjustment, BA), and downloaded to the follower for AR pose update optimization.

[0041] like Figure 4 The diagram shown illustrates the specific workflow of the AR dynamic update module. This workflow includes the following steps:

[0042] Input the AR pose in the placer's world coordinates. The relative positional relationship between two keyframes obtained from the adaptive unloading module is represented by the rotation matrix R and the translation vector t. The pose of the keyframe that was unloaded to the server is represented by the transformation matrix T. pw The pose of the keyframes from which the follower is unloaded onto the server is represented by the rotation matrix R. fw and translation vector t fw The current pose of the follower is represented by a rotation matrix. and translation vector express;

[0043] For each pose described above, world coordinate system, camera coordinate system, and pixel coordinate system transformations are performed, and the real-time pose of the follower at all three ends is calculated using the corresponding matrices. The formula is as follows:

[0044]

[0045] Where K is the camera intrinsic parameter, and the superscript T indicates transpose.

[0046] Output the AR pose coordinates in the follower coordinate system.

[0047] like Figure 5The diagram shown is an adaptive unloading flowchart. This process includes the following steps:

[0048] Camera frames are input from the mobile devices at the placer 1 and follower 2 respectively.

[0049] Determine whether the mobile devices at the placer 1 and follower 2 have received frame drop information from the edge server: if yes, proceed to the next step; if not, it means that the camera frame does not meet the conditions for being a key frame, so the camera frame is directly dropped; this determination is to avoid network congestion on the uplink.

[0050] To determine whether the current frame is significantly different from the most recently unloaded frame, we specifically compare the number of matching feature point pairs between the current frame and the most recently unloaded keyframe. If the comparison result is less than a threshold T... feature If the current field of view of the mobile devices at both the placer 1 and follower 2 has changed significantly, then proceed directly to the next step; otherwise, the camera frame does not meet the conditions for being a keyframe and is discarded. This judgment is made to save communication and computation costs.

[0051] This camera frame is offloaded as a keyframe to the edge server.

[0052] In summary, the advantages of the edge-assisted multi-user mobile augmented reality system of the present invention are as follows:

[0053] (1) Based on the high-precision positioning provided by SLAM, and equipped with edge collaborative optimization, AR dynamic update strategy and adaptive unloading strategy, this system can create an accurate and real-time multi-user MAR system.

[0054] (2) Compared with edge cloud-assisted and point-to-point design schemes, this system designs a parallel workflow of point-to-point fast recovery rendering and edge collaborative optimization to ensure the real-time performance of the system.

[0055] (3) This system combines emerging edge computing and deep learning-based target detection technology, making it possible to solve computationally intensive tasks on mobile devices and understand contextual information in multi-user MAR environments;

[0056] (4) This system has strong development potential. Augmented reality is both a way of experiencing the metaverse like virtual reality (VR) and a platform with location and background in the metaverse. Multi-user MAR allows multiple users to interact, cooperate, or compete with each other in the MAR system, and it is expected to provide immersive experiences in the fields of entertainment, education, and healthcare.

Claims

1. A multi-user mobile augmented reality system based on edge assistance, based on a network environment consisting of placers, servers, and followers, characterized in that, The system includes a placer adaptive unloading module on the placer side, an edge collaborative optimization module on the server side, and a follower adaptive unloading module, an AR dynamic update module, and a relative pose estimation module on the follower side; wherein: The placer adaptive unloading module selects the placer's keyframe, associated point cloud, and the placer's current pose, and unloads it to the server; and connects to the relative pose estimation module to solve a rough AR pose through relative pose estimation and send it to the follower. The edge collaboration optimization module obtains more robust feature matching point pairs between the placer and the follower through a strict loose matching algorithm, thereby obtaining a more accurate relative positional relationship between the two mobile devices, and continuously optimizing the AR pose of the follower. The specific process of the strict loose matching algorithm includes: inputting feature points on keyframes; determining whether the feature points are feature points on relatively static objects; if so, performing loose matching; if not, performing strict matching; and outputting feature matching point pairs. The follower adaptive unloading module selects the follower's keyframes, associated point clouds, and the follower's current pose, and unloads them to the server. The AR dynamic update module handles the AR pose dynamic update of the follower end to adapt to changes in time and space.

2. The edge-assisted multi-user mobile augmented reality system as described in claim 1, characterized in that, Keyframes follow two selection principles: Principle 1: Select the frame at the moment when the mobile device receives the dropped message from the edge server as the keyframe; Principle 2: Select the frame when the current frame is significantly different from the most recently uninstalled frame as the keyframe.

3. The edge-assisted multi-user mobile augmented reality system as described in claim 1, characterized in that, The edge collaborative optimization module includes keyframe preprocessing. It takes keyframes of both the placer and the follower as input, performs object detection on both keyframes using the YOLOv5x object detection algorithm, and outputs the object type, detection confidence score, and their two-dimensional bounding boxes, thus obtaining the environmental information contained in the keyframes. It then compares the object type of the follower with that of the placer; if the matching rate is greater than the established threshold T... match If the two unloading keyframes are in the same environment, it indicates that they are in the same environment; otherwise, these two frames are discarded and this information is fed back to the mobile device.

4. The edge-assisted multi-user mobile augmented reality system as described in claim 1, characterized in that, The edge collaborative optimization module includes feature point preprocessing, which inputs relevant point cloud information and the poses of the placer and follower keyframes to perform dynamic feature point culling, and outputs filtered relatively static feature points.

5. The edge-assisted multi-user mobile augmented reality system as described in claim 1, characterized in that, The specific process of the AR dynamic update module is as follows: Input the AR pose in the placer's world coordinates. The relative positional relationship between two keyframes obtained from the edge collaborative optimization module is represented by the rotation matrix R and the translation vector t. The pose of the keyframes unloaded by the placer to the server is represented by the transformation matrix. The pose of the keyframes representing the follower unloading to the server is represented by a rotation matrix. and translation vector The current pose of the follower is represented by a rotation matrix. and translation vector express; For each pose described above, world coordinate system, camera coordinate system, and pixel coordinate system transformations are performed, and the real-time pose of the follower at all three ends is calculated using the corresponding matrices. The formula is as follows: ； ； ； ； Where K is the camera intrinsic parameter, and the superscript T indicates transpose; Output the AR pose coordinates in the follower coordinate system. .

6. The edge-assisted multi-user mobile augmented reality system as described in claim 1, characterized in that, The process of adaptive unloading of modules by either the placer or the followers is as follows: Camera frames are input from the mobile devices on both the placement and follower ends. Determine whether the mobile devices on the placement end and the follower end have received frame drop information from the edge server: if yes, proceed to the next step; if not, it means that the camera frame does not meet the conditions for being a keyframe, so the camera frame is directly dropped; this determination is to avoid network congestion on the uplink. To determine whether the current frame is significantly different from the most recently unloaded frame, we specifically compare the number of matching feature point pairs between the current frame and the most recently unloaded keyframe. If the comparison result is less than a threshold T... feature This indicates that the current field of view of the mobile devices on both the placement and follower sides has changed significantly. Therefore, proceed directly to the next step; Otherwise, if a camera frame does not meet the conditions for being a keyframe, the camera frame is discarded directly. This decision is made to save on communication and computing costs; This camera frame is offloaded as a keyframe to the edge server.

7. A multi-user mobile augmented reality method based on edge-assisted technology, implemented using the multi-user mobile augmented reality system based on edge assistance as described in claim 1, characterized in that... This method Includes the following steps; Step 1: Perform adaptive unloading processing for the placer. On the one hand, select the keyframes of the placer, the associated point cloud, and the pose of the placer at this time to form a packet file on the placer side and unload it to the server. On the other hand, perform system initialization and use relative pose estimation to solve a rough AR pose and send it directly to the mobile device on the follower side. In addition, follower adaptive setup and offloading processing is performed, that is, the keyframe of the current follower, the associated point cloud and the current pose of the follower are selected to form a packet file on the follower side and offloaded to the server. Step 2: Perform edge collaborative optimization processing. Obtain feature matching point pairs through a strict loose matching algorithm. Calculate the relative positional relationship R and t between the placer and the follower using the feature matching point pairs, where R represents the rotation matrix and t represents the translation vector. This allows for understanding the contextual information in the environment and obtaining accurate scene recognition results. Step 3: Perform AR dynamic update processing, which means reusing the pose of the keyframe uploaded by the follower to the server and the current pose, and dynamically updating the rough AR pose in Step 1 to adapt the edge collaborative optimization results to changes in time and space, continuously optimizing the AR pose of the follower, and obtaining real-time and accurate AR pose dynamic update results to input the mobile device of the follower.

Images