A cloud rendering-based multi-person AR interaction method

Abstract

The application discloses a kind of multi-person AR interaction methods based on cloud rendering, comprising the following steps: user wears lightweight AR glasses, and is joined into interactive system by multi-person AR interaction application;Multi-person AR interaction application sends the request of establishing P2P connection to signaling server, and connection is established with cloud server after request is accepted;AR glasses shoot video frame and send to cloud server, and cloud searches and matches received video frame and pre-stored image library and reconstructed dense point cloud.The application completes rendering and visual positioning and other computing tasks in cloud, solves the high demand of existing scheme to device hardware, greatly improves the resource utilization of server and the cross-platform of system;Adopt end cloud fusion rendering algorithm, rendering task can be adaptively scheduled, so as to improve the upper limit of the number of users that system can accommodate.The application not only can provide deep immersive 3D interaction experience, wear comfortable, experience effect is very good.

Description

Technical Field

[0001] This invention relates to the field of multi-user AR interaction technology, and more specifically, to a multi-user AR interaction method based on cloud rendering. Background Technology

[0002] Augmented reality (AR) technology enables interactive effects that blend the virtual and real worlds, providing consistent shared and collaborative services for multiple users. In many existing multi-user AR interaction solutions, the core AR functions and rendering tasks are primarily implemented on the client side, with the server acting as a communication medium between multiple terminals. This means that the main computational tasks of the entire system are performed by the terminals. These computational tasks mainly include: 3D rendering, information transmission, human-computer interaction, and AR tracking. Based on the terminal platform the system is designed for, it can be categorized into the following three types:

[0003] 1) Application-oriented multi-user interactive AR: This type of system uses applications as carriers and mobile devices such as mobile phones or tablets to provide a synchronous and consistent AR interactive experience for multiple users.

[0004] 2) Web-based multi-user interactive AR: In this mode, the browser uses input information collected from the phone or tablet's GPS, gyroscope, camera, etc., to calculate the device's current pose information, thereby providing core AR functionality. Furthermore, a P2P or WS connection can be established between the browser and the server to enable synchronous interaction among multiple users.

[0005] 3) Multi-user interactive AR for head-mounted AR devices: Users interact with each other in AR through head-mounted AR devices, providing a better immersive 3D experience. All rendering and AR calculations are completed on the terminal device.

[0006] However, each of the aforementioned existing carrier schemes has its own drawbacks:

[0007] 1) AR based on applications: In AR applications based on devices such as mobile phones or tablets, users' hands are bound by the device screen, which limits their ability to interact with the real space and thus affects the interactive experience; in addition, users cannot get a deeply immersive experience when facing a 2D screen, lack a sense of realism, and cannot achieve a good virtual-real fusion effect.

[0008] 2) Web-based multi-user interactive AR: Due to browser limitations on access to relevant sensors, the AR effects in this mode often cannot compare with those of application-based or head-mounted AR devices.

[0009] 3) Head-mounted AR devices: Traditional head-mounted AR devices may suffer from poor comfort. Lightweight AR helmets, on the other hand, have relatively weak computing power and cannot support the computing power requirements of complex multi-terminal shared collaborative AR systems.

[0010] Another drawback of existing technical solutions is that the main AR functions and rendering tasks are implemented on the client side, while the server mainly acts as a medium for communication between multiple terminals. This raises several issues:

[0011] 1) Hardware requirements: In existing multi-person AR interaction solutions, the main computing tasks, including 3D rendering, information transmission, interaction synchronization and AR positioning, need to be completed by the terminal device, which places high demands on the terminal hardware (such as GPU, CPU and memory).

[0012] 2) Resource sharing: In the existing solution, the server's main task is to forward communication data, and the server's computing resources have not been fully utilized;

[0013] 3) Cross-platform compatibility: Since AR headsets may use different operating systems, such as HoloLens using the UWP platform and NrealLight using the Android platform, existing solutions require the development of different clients for different terminal devices, and the workload of development increases with the increase in the types of terminal devices.

[0014] There is currently no effective solution to the above problems. Summary of the Invention

[0015] To address the aforementioned technical problems in related technologies, this invention proposes a multi-user AR interaction method based on cloud rendering. This method solves the problems of excessively high hardware requirements for devices, insufficient computing power of lightweight head-mounted AR devices, poor rendering effects, and inability to effectively support the heavy rendering tasks in multi-user AR interaction systems in existing technical solutions. It overcomes the aforementioned shortcomings of existing technologies.

[0016] To achieve the above-mentioned technical objectives, the technical solution of the present invention is implemented as follows:

[0017] A multi-user AR interaction method based on cloud rendering includes the following steps:

[0018] S1 users wear lightweight AR glasses and join the interactive system through a multi-person AR interactive application;

[0019] The S2 multi-person AR interactive application sends a P2P connection establishment request to the signaling server. After the request is accepted, a connection is established with the cloud server.

[0020] The S3 AR glasses capture video frames and send them to the cloud server. The cloud retrieves and matches the received video frames, as well as the pre-stored image library and the reconstructed dense point cloud, and obtains the 6DoF head pose of the user in the current real space through the pose estimation algorithm.

[0021] The S4 AR glasses acquire and send the camera's real-time transformation matrix through local binocular SLAM. The cloud server then transforms the user's virtual camera in the cloud rendering scene to the corresponding pose based on the received transformation matrix.

[0022] S5 obtains the number of triangles and texture pixels of the 3D model to be rendered in the scene, and gets the current number of users from the cloud.

[0023] The S6 cloud server uses a tabu search algorithm to adaptively allocate rendering tasks, achieving load balancing of mixed rendering tasks between AR glasses and cloud rendering servers;

[0024] The S7 cloud server renders the virtual image from the camera and returns it to the user as a video stream. The AR glasses then merge the results of the remote rendering with the results of the local rendering and output them.

[0025] When a user interacts with the scene model, the S8 acquires the mutex lock of the target model, receives interaction command information through a local copy of the scene model, and sends the status information to the cloud server for synchronization to achieve multi-person AR interaction.

[0026] Furthermore, in step S2, the multi-user AR interactive application establishes a WebSocket connection with the signaling server, and the multi-user AR interactive application negotiates a P2P connection with the cloud rendering host through the WebSocket connection; the P2P connection is used by the rendering host to transmit the rendered video stream and audio stream to the client, and a DataChannel is established at the same time to transmit user interaction data, camera pose data and text information.

[0027] Furthermore, the specific steps of step S3 are as follows:

[0028] S301: Use an RGB camera to capture a set of images of the actual scene from all angles to obtain a collection of pictures in advance;

[0029] S302: Input the image set into the 3D reconstruction algorithm, and construct a sparse or dense point cloud model through feature extraction, feature matching, product geometry, and triangulation;

[0030] S303: The cloud server uses an image retrieval-based visual positioning algorithm to extract global and local features based on the RGB frames sent by the client. The global features are used to perform image retrieval in the image set. Local features are used to perform 2D to 3D matching in the K most similar images retrieved. The camera pose is calculated using the 3D point pose information in the point cloud model.

[0031] S304: Based on the camera pose obtained in step S31, the cloud server places the user camera in the virtual scene of the cloud rendering host at the corresponding position in the point cloud model coordinate system, and runs step S33 to obtain the camera pose for reset and update at a certain frequency.

[0032] Furthermore, the specific process of step S303 is as follows:

[0033] S3031 creates a camera settings parameter class, sets the pixel width and height values ​​of the camera frame, sets the encoding format to RGBA32 format, and finally sets the blending mode to RGB only mode;

[0034] The S3032 runs the camera capture function in a new thread, passing in the settings parameters and specifying the callback function to be executed after the camera capture is complete. Once the child thread finishes capturing the image, it passes the captured image as a bit array to the callback function. The callback function then sends the image to the server, triggering the rendering host's localization event. The rendering host executes an image retrieval-based visual localization algorithm to extract global features from the image for image retrieval against an image library. For multiple retrieved images, it obtains their corresponding 3D point clouds and uses SIFT to extract local features and 2D feature points. The RANSAC-PnP algorithm is then used to match the 2D feature points and 3D point clouds to calculate the camera pose of the image frame.

[0035] S3033 triggers a positioning event, returning the positioning result to the client in JSON format; if an exception occurs in the positioning algorithm, the exception is captured, and a positioning error event is triggered, sending the error information to the client and applying the positioning result to the camera in the virtual scene of the cloud rendering host.

[0036] Furthermore, the specific steps of step S4 are as follows:

[0037] S401: The AR glasses obtain the transformation matrix T of the camera at different times through a local binocular SLAM algorithm, where T contains the camera's rotation matrix and translation vector;

[0038] S402: Convert the transformation matrix into a quaternion and a displacement vector, and send the rotation matrix, displacement vector, quaternion and displacement vector to the cloud server;

[0039] S403: After receiving the camera's displacement vector, the cloud performs the following spatial transformation:

[0040]

[0041] Among them, [X B Y B Z B ] T For the desired virtual space coordinates, [X A Y A Z A ] T The spatial coordinates after transformation by transformation matrix T, T X T y T z Let W be the translation parameter from coordinate system B to coordinate system A. X W y W z is the rotation parameter from coordinate system B to coordinate system A, m is the scaling parameter from coordinate system B to coordinate system A, T is the transformation matrix from coordinate system B to coordinate system A, and K' is an intermediate variable. The formula for calculating K' is as follows:

[0042]

[0043] S404: The cloud uses quaternion multiplication to transform the coordinate system for the angle. The calculation formula is as follows:

[0044] q c =q a q Δ

[0045] The quaternion representing the head rotation after the user's movement is q. a Its initial quaternion representation in the rendering host is q. c q Δ It is the change in the rotation quaternion, q Δ Let be the rotational change from a to c.

[0046] Furthermore, the specific steps of step S6 are as follows:

[0047] S601: When a new user joins, a tabu search-based decision algorithm runs in the cloud, with input parameters including the current number of users and existing decision options;

[0048] S602: The decision algorithm constructs a neighborhood based on the current decision options using search operators;

[0049] S603: The decision algorithm traverses all possible decision schemes in the current domain, calculates the evaluation function value of each scheme, and selects the decision scheme with the optimal function value;

[0050] S604: If the optimal decision solution does not appear in the tabu list in this iteration or the decision solution meets a specific exception criterion, then add it to the tabu list as the result of this iteration; if the optimal decision solution does not meet the above conditions, then select the second-best decision solution that meets the conditions as the result of this iteration and add it to the tabu list.

[0051] S605: The above steps continue to iterate until a specific termination condition is met, at which point the final decision is returned. At the same time, the cloud assigns rendering tasks to each terminal device according to the final decision and notifies all terminal applications of the task assignment results.

[0052] Furthermore, the process of the tabu search-based edge-cloud fusion rendering algorithm in step S6 is as follows:

[0053] S611 checks if the current frame rate is below 60. If it is, no action is taken. When the frame rate is below 60, the initial solution is set to X. last If the tabu table Table is empty, then the historical best solution X is... min Set as the current solution;

[0054] S612 begins the iterative search process: In each search process, starting from the current solution X... i We begin by constructing a solution X that does not repeat the taboo list. i Given a domain A, find the optimal solution X for the evaluation function within that domain. j And add the path to the taboo table, then add the solution X. i Updated to solve X j Then, if the evaluation function of the new solution is better, then the global optimal solution X is obtained. min The new interpretation of X j The solution is replaced; otherwise, the global optimal solution remains unchanged.

[0055] A new solution has been found for S613. j Then, return to step S611, continue to judge the loop condition and continue to execute. When the number of iterations reaches the termination condition or the optimal solution is found, the algorithm ends.

[0056] Furthermore, the specific steps of step S8 are as follows:

[0057] S81: The cloud uses the bounding box algorithm to calculate the bounding box information of the object that the user interacts with;

[0058] S82: The cloud performs coordinate transformation from virtual space to real space on the bounding box information and sends the transformed bounding box information to various terminal devices;

[0059] S83: Each terminal application constructs a corresponding collision detection body in the real space based on the received bounding box information;

[0060] S84: When the user of the AR glasses interacts with the collision detection object through gestures or a laser handle, the terminal device sends an interaction request to the cloud;

[0061] S85: After receiving an interaction request, the cloud acquires the mutex lock of the interaction object. If the acquisition fails, a message indicating that the request failed is returned; if the acquisition succeeds, a message indicating that the request succeeds is returned.

[0062] S86: Users interact with collision detection objects in the real space and send the corresponding interaction information to the cloud;

[0063] S87: The cloud receives the operation information of the collision detection object, converts the operation information from the real space back to the virtual space, and applies the operation information to the interactive objects in the virtual space.

[0064] The beneficial effects of this invention are as follows: By completing computational tasks such as rendering and visual positioning in the cloud, this invention solves the high hardware requirements of existing solutions, greatly improving server resource utilization and system cross-platform compatibility. Employing an edge-cloud fusion rendering algorithm, it can adaptively schedule rendering tasks, thereby increasing the upper limit of the number of users the system can accommodate. This invention not only provides a deeply immersive 3D interactive experience, but the device is also very lightweight, comfortable to wear, and offers an excellent user experience. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 This is a flowchart of a multi-user AR interaction method based on cloud rendering according to an embodiment of the present invention;

[0067] Figure 2 This is a schematic diagram of a cloud rendering viewpoint roaming method based on a cloud rendering multi-person AR interaction method according to an embodiment of the present invention;

[0068] Figure 3 This is a schematic diagram of the edge-cloud fusion adaptive rendering task scheduling algorithm of the multi-person AR interaction method based on cloud rendering according to an embodiment of the present invention. Detailed Implementation

[0069] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0070] like Figure 1-3 As shown in the embodiments of the present invention, in order to solve the problems of excessively high hardware requirements of existing equipment, insufficient computing power of lightweight head-mounted AR devices, poor rendering effects, and inability to effectively support the heavy rendering tasks in multi-person AR interaction systems, the present invention provides a multi-person AR interaction method and system based on cloud rendering, including the following steps:

[0071] S11: First, multiple users wear lightweight AR glasses and join the interactive system through the multi-user AR interactive application provided by this invention.

[0072] S12: The application sends a P2P connection establishment request to the signaling server. After the request is accepted, a connection is established with the cloud server.

[0073] S13: The AR glasses capture video frames and send them to the cloud server. The cloud retrieves and matches these video frames, along with a pre-stored image library and reconstructed dense point clouds, and then uses a pose estimation algorithm to obtain the user's 6DoF head pose in the real-world space.

[0074] S14: The AR glasses acquire and send the camera's real-time transformation matrix through local binocular SLAM. The cloud server then transforms the user's virtual camera in the cloud rendering scene to the corresponding pose based on the received transformation matrix.

[0075] S15: For the 3D model that needs to be rendered in the scene, obtain the number of triangles and texture pixels of the model, and obtain the current number of users from the cloud.

[0076] S16: Based on the above factors and the tabu search algorithm, the cloud server performs adaptive rendering task allocation, thereby achieving load balancing of mixed rendering tasks between AR glasses and the cloud rendering server.

[0077] S17: The cloud server renders the virtual image from the camera and returns it to the user as a video stream. The AR glasses then merge the results of the remote rendering with the results of the local rendering and output them.

[0078] S18: When a user interacts with the scene model, they first need to acquire the mutex lock of the target model. Once successfully acquired, they can receive interaction command information through a local copy of the scene model, and then send the state information to the cloud server for synchronization to achieve multi-person AR interaction.

[0079] Step S13 includes the following sub-steps:

[0080] S31: Use an RGB camera to take pictures of the actual scene from all angles to obtain a set of photos.

[0081] S32: Input the photo set into the 3D reconstruction algorithm, and construct a sparse or dense point cloud model through steps such as feature extraction, feature matching, product geometry, and triangulation.

[0082] S33: The cloud-based visual positioning algorithm uses image retrieval to extract global and local features based on the RGB frames sent by the client. Global features are used for image retrieval in the image set of S31. Among the K most similar images retrieved, local features are used for 2D to 3D matching. Finally, the camera pose is calculated using the 3D point pose information in the point cloud model.

[0083] S34: Based on the camera pose obtained in S31, the cloud server places the user's camera in the virtual scene of the cloud rendering host at the corresponding position in the point cloud model coordinate system, and runs step S33 at a certain frequency to obtain the camera pose for reset and update.

[0084] Step S14 includes the following sub-steps:

[0085] S41: The AR glasses obtain the camera's transformation matrix T at different times using a local binocular SLAM algorithm. T is a 4×4 matrix containing the camera's rotation matrix and translation vector.

[0086] S42: Convert the transformation matrix in S41 into a quaternion and a displacement vector, and send these 7 numbers to the cloud server.

[0087] S43: After receiving the camera's displacement vector, the cloud performs the following spatial transformation:

[0088]

[0089] Among them, [X B Y B Z B ] T For the desired virtual space coordinates, [X A Y A Z A ] T The spatial coordinates after transformation by transformation matrix T, T X T y T z Let W be the translation parameter from coordinate system B to coordinate system A. X W yWz refers to the rotation parameter from coordinate system B to coordinate system A, m refers to the scaling parameter from coordinate system B to coordinate system A, T refers to the transformation matrix from coordinate system B to coordinate system A, and K' is an intermediate variable. The formula for calculating K' is as follows:

[0090]

[0091] S44: The cloud performs coordinate system transformations for angles using quaternion multiplication, with the following formula:

[0092] q c =q a q Δ

[0093] The quaternion representing the head rotation after the user's movement is q. a Its initial quaternion representation in the rendering host is q. c . q Δ It is a rotational quaternion, describing the rotational change from a to c.

[0094] The S16 further includes:

[0095] S61: When a new user joins, a tabu search-based decision algorithm runs in the cloud, with input parameters including the current number of users and existing decision options.

[0096] S62: The decision algorithm uses search operators to construct a neighborhood on the current decision options.

[0097] S63: The decision algorithm iterates through all possible decision schemes in the current domain, calculates the evaluation function value of each scheme, and then selects the decision scheme with the optimal function value.

[0098] S64: If the optimal decision solution does not appear in the tabu list in this iteration, or if the decision solution meets a specific exception criterion, then take it as the result of this iteration and add it to the tabu list.

[0099] S65: If the optimal decision scheme does not satisfy the conditions in S64, then select the suboptimal decision scheme that satisfies the conditions as the result of this iteration and add it to the tabu list.

[0100] S66: The above steps will continue to iterate until a specific termination condition is met, at which point the final decision solution is returned. Then, the cloud will allocate rendering tasks to various terminal devices according to this solution and notify all terminal applications of the task allocation results.

[0101] Step S18 further includes:

[0102] S81: The cloud uses the bounding box algorithm to calculate the bounding box information of the object that the user wants to interact with.

[0103] S82: The cloud performs coordinate transformation from virtual space to real space on the bounding box information, and then sends the transformed bounding box information to various terminal devices.

[0104] S83: Each terminal application constructs a corresponding collision detection volume in the real space based on the received bounding box information.

[0105] S84: When the user of the AR glasses interacts with the collision detection object using gestures or a laser handle, it indicates that the user wishes to interact with the object. At this time, the terminal device will send an interaction request to the cloud.

[0106] S85: After receiving an interaction request, the cloud attempts to acquire the mutex lock of the interaction object. If the acquisition fails, a message indicating a request failure is returned; if the acquisition succeeds, a message indicating a request success is returned.

[0107] S86: Users perform actions such as rotation and movement on collision detection objects in real space, and then send this operation information to the cloud.

[0108] S87: After receiving the collision detection information from the cloud, it converts the object from the real space back to the virtual space, and then applies these operations to the interactive objects in the virtual space.

[0109] To facilitate understanding of the above technical solutions of the present invention, the following detailed description of the above technical solutions of the present invention will be provided through specific usage methods.

[0110] In practical use, the multi-user AR interaction method based on cloud rendering according to the present invention includes the following steps:

[0111] S11: Multiple users wearing lightweight AR glasses join the interactive system through the multi-user AR interactive application provided by this invention.

[0112] S12: The application sends a request to the signaling server to establish a peer-to-peer (P2P) connection, and establishes a connection with the cloud server after receiving a response.

[0113] S13: The AR glasses capture video frames and send them to the cloud server. The cloud server retrieves and matches these frames against a pre-stored image library and a reconstructed dense point cloud, and uses a pose estimation algorithm to obtain the user's 6DoF head pose in the real-world space.

[0114] S14: The AR glasses acquire and send the real-time camera transformation matrix through local binocular simultaneous localization and mapping (SLAM). The cloud server transforms the user's virtual camera in the cloud-rendered scene to the corresponding pose based on the received transformation matrix.

[0115] S15: For a 3D model that needs to be rendered, obtain the number of triangles and texture pixels of the model, and obtain the current number of users from the cloud.

[0116] S16: The cloud server performs adaptive rendering task allocation based on the above factors and the tabu search algorithm to achieve load balancing of hybrid rendering tasks between AR glasses and cloud rendering servers.

[0117] S17: The cloud server renders the image from the virtual camera's perspective and returns it to the user's camera view as a video stream. The AR glasses merge the remote rendering results with the local rendering results for output.

[0118] S18: When a user interacts with the scene model, the mutex lock of the target model is acquired first. After successfully acquiring the lock, the interaction command information is received through a local copy of the scene model, and then its state information is sent to the cloud server for synchronization to realize multi-person AR interaction.

[0119] Specifically:

[0120] In the S11, users wear AR glasses and join the interactive system through the provided application, supporting interaction methods such as gestures or laser handles.

[0121] In S12, the application establishes a WebSocket connection with the signaling server, through which it negotiates a P2P connection with the cloud rendering host. The application sends an OfferSDP (SessionDescription Protocol) to the signaling server to establish the connection, which is then forwarded to the rendering host. After a series of negotiations, the rendering host and the client are able to establish a P2P connection. This connection is used by the rendering host to transmit rendered video and audio streams to the client, and a DataChannel is also established to transmit user interaction data, camera pose data, and text information.

[0122] In S13, the AR glasses capture video frames and send them to a cloud server. The cloud server retrieves and matches these frames against a pre-stored image library and a reconstructed dense point cloud, using a pose estimation algorithm to obtain the user's current spatial pose. The system uses Colmap to perform sparse point cloud 3D reconstruction of the real-world scene, and the generated point cloud map contains the pose information of each point cloud in real space.

[0123] After the point cloud map is built, the user-end initialization captures images of the real-world scene. When acquiring RGB camera frames from the AR glasses, the camera API within the AR glasses is used. First, a camera settings parameter class is created, setting the pixel width and height values ​​of the camera frames and setting the encoding format to RGBA32. Finally, the blending mode is set to RGB-only, ensuring that the captured image only includes the real-world scene and excludes the virtual scene. Then, a camera capture function is run in a new thread, passing in the settings parameters and specifying a callback function to execute after the camera capture is complete. After the child thread completes the camera capture, the captured image is passed to the callback function as a bit array. This function then sends the image to the server, triggering a localization event on the rendering host. The rendering host executes an image retrieval-based visual localization algorithm, extracting global features from the image for image retrieval against an image library. For multiple retrieved images, their corresponding 3D point clouds are obtained, and SIFT is used to extract local features and 2D feature points from the images. Finally, the RANSAC-PnP algorithm is used to match the 2D feature points and 3D point clouds to calculate the camera pose of the image frames. Finally, a location event is triggered, and the location result is returned to the client in JSON format. If an exception occurs in the location algorithm, the exception is captured, a location error event is triggered, the error information is sent to the client, and the location result is applied to the camera in the virtual scene of the cloud rendering host.

[0124] In S14, the AR glasses run a local binocular visual inertial odometry calculation in real time and feed the pose results back to the cloud server. The cloud server then updates the camera pose to enable lens movement and rotation. For example... Figure 2 The viewpoint roaming diagram shown first obtains the user's initial pose information through the visual positioning module in step S13, and establishes and saves an initial coordinate system using the x, y, and z unit vectors of the initial pose. The terminal records the pose information of the AR headset when sending RGB camera frames. Whenever the user moves or rotates the AR headset, the transformation matrix relative to the initial pose is calculated and converted into quaternions and a 3D position transformation. The terminal serializes these quaternions and 3D position transformations into JSON using a bitstream and sends it to the cloud. The cloud receives these quaternions and 3D position transformations and deserializes them. The rotation angle of the current camera is calculated using the right multiplication of the quaternions, and the 3D position transformation is converted from the camera coordinate system to the world coordinate system and added to the current camera position. Then, the calculated quaternions and position are assigned to the Transform class of the user's camera object in the virtual space.

[0125] In S15, for 3D models that need to be rendered in the virtual scene, the Unity3D engine is used to obtain the number of triangles (tris) and texture pixel values ​​(pixs) of the model, and the number of triangles and texture pixel values ​​of all models are saved, and the number of users in the current system is recorded.

[0126] In S16, when a new terminal joins the system, the system makes a decision using an edge-cloud fusion algorithm. The server maintains a decision data table containing m 3D models and n terminal users from the virtual scene. A one-dimensional array X represents the decision schemes of all fusion positioning algorithms, and the solution space of X is 2^n. m The power of X, where X j The value is either 0 or 1, where 0 indicates that the j-th model is rendered in the cloud, and 1 indicates that the j-th model is rendered on the user's terminal. Using f... Total (X) represents the total rendering frame rate of the system under the current decision-making method, using RD Total (X) represents the total interaction latency of the system. The evaluation function F(X) represents the current user experience quality of the system, and its function value is obtained from the variables in S15.

[0127] like Figure 3 As shown, the process of the tabu search-based edge-cloud fusion rendering algorithm is as follows: First, it is determined whether the current frame rate is lower than 60. If it is lower than 60, no processing is performed. When the frame rate is lower than 60, the initial solution is set to X. last If the tabu table Table is empty, then the historical best solution X is... min Set it as the current solution. Then begin a multi-iterative search process. In each search process, start from the current solution X. i We begin by constructing a solution X that does not repeat the taboo list. i The domain is A. Then, the optimal solution X for the evaluation function is found within the domain. j And add the path to the taboo table. Then, in the solution X... i Updated to solve X j Then, if the evaluation function of the new solution is better, then the global optimal solution X is obtained. min The new interpretation of X j The solution is replaced; otherwise, the global optimal solution remains unchanged. Although the new solution X... j It may worsen, but as the search space expands, it still helps to escape local optima. Find a new solution X. j Then, return to the first step of the iteration, check the loop condition, and continue execution. The algorithm ends when the number of iterations reaches the termination condition or the optimal solution is found.

[0128] In S17, the primary task of cloud rendering is to render 3D models. In 3D rendering, the rendering task is completed by the GPU, utilizing a rendering pipeline in computer graphics. This pipeline fully leverages the GPU's parallel computing capabilities, thereby accelerating the rendering process. After the rendering host renders the 3D scene, the rendering result needs to be transmitted to the AR terminal as a video stream. This invention uses WebRTC technology for terminal-to-terminal P2P streaming of the rendered image stream. The rendered texture is in RGBA format, therefore it needs to be converted to the I420 format accepted by the encoder using the videoconvert element. After conversion, each frame is approximately 6.9MB. Direct transmission would put significant pressure on bandwidth, causing network transmission to fail and resulting in a drop in frame rate. Therefore, after establishing a P2P connection, a compression encoding module processes the input rendered texture (60fps), compressing it into a video stream, which is then transmitted to the client via the WebRTC protocol. This system uses H.264 to encode and compress the input rendered texture, and also supports other encoders such as VP8, VP9, ​​and AV1X. Finally, the generated compressed bitstream is packaged into RTP packets, encrypted, and sent to the client using the WebRTC protocol.

[0129] In S18, when a user interacts with the scene model, they first need to acquire the mutex lock of the target model. After successfully acquiring the lock, they receive interaction command information through a local copy of the scene model, and then send its state information to the cloud server for synchronization, thereby achieving the purpose of multi-person AR interaction. This invention converts the bounding box information of the target object model in the cloud-rendered scene into a pose in the user's spatial coordinate system, and sends the pose information and bounding box information to the terminal. The terminal constructs a corresponding collision detection bounding box on the corresponding object in the local 2D video frame to detect the terminal's interaction commands. During the interaction, the transformation information of the target object's bounding box is synchronized to the server in real time and broadcast to other terminals for sharing. This system generates a 3D target object bounding box in the target model area through pose calculation in the 2D video frame for receiving the terminal's interaction commands and performing remote interaction in cloud rendering.

[0130] In summary, by utilizing the technical solution of this invention, rendering and visual positioning computational tasks are completed in the cloud, thus solving the high hardware requirements of existing solutions and greatly improving server resource utilization and system cross-platform compatibility. The adoption of an edge-cloud fusion rendering algorithm allows for adaptive scheduling of rendering tasks, thereby increasing the upper limit of the number of users the system can accommodate. This invention not only provides a deeply immersive 3D interactive experience, but the device is also very lightweight, comfortable to wear, and offers an excellent user experience.

[0131] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-user AR interaction method based on cloud rendering, characterized in that, Includes the following steps: S1 users wear lightweight AR glasses and join the interactive system through a multi-person AR interactive application; The S2 multi-person AR interactive application sends a P2P connection establishment request to the signaling server. After the request is accepted, a connection is established with the cloud server. The S3 AR glasses capture video frames and send them to the cloud server. The cloud retrieves and matches the received video frames, as well as the pre-stored image library and the reconstructed dense point cloud, and obtains the 6DoF head pose of the user in the current real space through the pose estimation algorithm. The S4 AR glasses acquire and send the camera's real-time transformation matrix through local binocular SLAM. The cloud server then transforms the user's virtual camera in the cloud rendering scene to the corresponding pose based on the received transformation matrix. S5 obtains the number of triangles and texture pixels of the 3D model to be rendered in the scene, and gets the current number of users from the cloud. The S6 cloud server uses a tabu search algorithm to adaptively allocate rendering tasks, achieving load balancing of mixed rendering tasks between AR glasses and cloud rendering servers; The S7 cloud server renders the virtual image from the camera and returns it to the user as a video stream. The AR glasses then merge the results of the remote rendering with the results of the local rendering and output them. When a user interacts with the scene model, the S8 acquires the mutex lock of the target model, receives interaction command information through a local copy of the scene model, and sends the status information to the cloud server for synchronization to achieve multi-person AR interaction.

2. The multi-user AR interaction method based on cloud rendering according to claim 1, characterized in that, In step S2, the multi-user AR interactive application establishes a WebSocket connection with the signaling server, and the multi-user AR interactive application negotiates a P2P connection with the cloud rendering host through the WebSocket connection. The P2P connection is used to transmit the rendered video and audio streams from the rendering host to the client, while a DataChannel is established to transmit user interaction data, camera pose data, and text information.

3. The multi-user AR interaction method based on cloud rendering according to claim 1, characterized in that, The specific steps of step S3 are as follows: S301: Use an RGB camera to capture a set of images of the actual scene from all angles to obtain a collection of pictures in advance; S302: Input the image set into the 3D reconstruction algorithm, and construct a sparse or dense point cloud model through feature extraction, feature matching, product geometry, and triangulation; S303: The cloud server uses an image retrieval-based visual positioning algorithm to extract global and local features based on the RGB frames sent by the client; Global features are used for image retrieval in the image set. Local features are used to perform 2D to 3D matching on the K most similar images retrieved, and the camera pose is calculated using the 3D point pose information in the point cloud model. S304: Based on the camera pose obtained in step S31, the cloud server places the user camera in the virtual scene of the cloud rendering host at the corresponding position in the point cloud model coordinate system, and runs step S33 to obtain the camera pose for reset and update at a certain frequency.

4. The multi-user AR interaction method based on cloud rendering according to claim 3, characterized in that, The specific process of step S303 is as follows: S3031 creates a camera settings parameter class, sets the pixel width and height values ​​of the camera frame, sets the encoding format to RGBA32 format, and finally sets the blending mode to RGB only mode; S3032 runs the camera capture function in a new thread, passes in the settings parameters, and specifies the callback function to be executed after the camera capture is completed. When the child thread finishes capturing the camera, it passes the captured image to the callback function in the form of a bit array. In the callback function, the image is sent to the server, triggering the positioning event of the rendering host. The rendering host executes a visual localization algorithm based on image retrieval to extract global features from the image for image retrieval against the image library. For multiple retrieved images, the corresponding 3D point clouds are obtained, and local features and 2D feature points in the images are extracted using SIFT. The RANSAC-PnP algorithm is used to match the 2D feature points and 3D point clouds to calculate the camera pose of the image frames. S3033 triggers a location event and returns the location result to the client in JSON format. If an anomaly occurs in the localization algorithm, the anomaly is captured, a localization error event is triggered, the error information is sent to the client, and the localization result is applied to the camera in the virtual scene of the cloud rendering host.

5. The multi-user AR interaction method based on cloud rendering according to claim 1, characterized in that, The specific steps of step S4 are as follows: S401: The AR glasses obtain the transformation matrix T of the camera at different times through a local binocular SLAM algorithm, where T contains the camera's rotation matrix and translation vector; S402: Convert the transformation matrix into a quaternion and a displacement vector, and send the rotation matrix, displacement vector, quaternion and displacement vector to the cloud server; S403: After receiving the camera's displacement vector, the cloud performs the following spatial transformation: Among them, [X B Y B Z B ] T For the desired virtual space coordinates, [X A Y A Z A ] T The spatial coordinates after transformation by transformation matrix T, T X T y T z Let W be the translation parameter from coordinate system B to coordinate system A. X W y W z is the rotation parameter from coordinate system B to coordinate system A, m is the scale parameter from coordinate system B to coordinate system A, T is the transformation matrix from coordinate system B to coordinate system A, and K′ is an intermediate variable. The formula for calculating K′ is as follows: S404: The cloud uses quaternion multiplication to transform the coordinate system for the angle. The calculation formula is as follows: q c =q a q Δ The quaternion representing the head rotation after the user's movement is q. a Its initial quaternion representation in the rendering host is q. c q Δ It is the change in the rotation quaternion, q Δ Let be the rotational change from a to c.

6. The multi-user AR interaction method based on cloud rendering according to claim 1, characterized in that, The specific steps of step S6 are as follows: S601: When a new user joins, a tabu search-based decision algorithm runs in the cloud, with input parameters including the current number of users and existing decision options; S602: The decision algorithm constructs a neighborhood based on the current decision options using search operators; S603: The decision algorithm traverses all possible decision schemes in the current domain, calculates the evaluation function value of each scheme, and selects the decision scheme with the optimal function value; S604: If the optimal decision solution does not appear in the tabu list in this iteration or the decision solution meets a specific exception criterion, then add it to the tabu list as the result of this iteration. If the optimal decision does not meet the above conditions, the suboptimal decision that meets the conditions is selected as the result of this iteration and added to the tabu list. S605: The above steps continue to iterate until a specific termination condition is met, at which point the final decision is returned. At the same time, the cloud assigns rendering tasks to each terminal device according to the final decision and notifies all terminal applications of the task assignment results.

7. The multi-user AR interaction method based on cloud rendering according to claim 1, characterized in that, The process of the tabu search-based edge-cloud fusion rendering algorithm in step S6 is as follows: S611 checks if the current frame rate is below 60. If it is, no action is taken. When the frame rate is below 60, the initial solution is set to X. last If the tabu table Table is empty, then the historical best solution X is... min Set as the current solution; S612 begins the iterative search process: In each search process, starting from the current solution X... i We begin by constructing a solution X that does not repeat the taboo list. i Given a domain A, find the optimal solution X for the evaluation function within that domain. j And add the path to the taboo table, then add the solution X. i Updated to solve X j Then, if the evaluation function of the new solution is better, then the global optimal solution X is obtained. min The new interpretation of X j The solution is replaced; otherwise, the global optimal solution remains unchanged. A new solution has been found for S613. j Then, return to step S611, continue to judge the loop condition and continue to execute. When the number of iterations reaches the termination condition or the optimal solution is found, the algorithm ends.

8. The multi-user AR interaction method based on cloud rendering according to claim 1, characterized in that, The specific steps of step S8 are as follows: S81: The cloud uses the bounding box algorithm to calculate the bounding box information of the object that the user interacts with; S82: The cloud performs coordinate transformation from virtual space to real space on the bounding box information and sends the transformed bounding box information to various terminal devices; S83: Each terminal application constructs a corresponding collision detection body in the real space based on the received bounding box information; S84: When the user of the AR glasses interacts with the collision detection object through gestures or a laser handle, the terminal device sends an interaction request to the cloud; S85: After receiving an interaction request, the cloud acquires the mutex lock of the interaction object. If the acquisition fails, a message indicating that the request failed is returned; if the acquisition succeeds, a message indicating that the request succeeds is returned. S86: Users interact with collision detection objects in the real space and send the corresponding interaction information to the cloud; S87: The cloud receives the operation information of the collision detection object, converts the operation information from the real space back to the virtual space, and applies the operation information to the interactive objects in the virtual space.

Images