Unity-based WeChat applet AR development framework and application thereof

Abstract

The invention provides a Unity-based WeChat applet AR (Augmented Reality) development framework and application thereof, and belongs to the technical field of applet development, and the Unity-based WeChat applet AR development framework adopts a'applet end-Unity runtime-algorithm layer-resource layer 'four-layer architecture. Comprising seven core modules: an applet development portal entry module, a Unity embedding and runtime container module, a self-developed SLAM and 6DoF tracking algorithm library module, a coordinate system alignment and AR space management module, a virtual resource editing, loading and management module, a terminal camera and virtual resource rendering fusion module and an applet and Unity mixed rendering interaction module. The invention aims to fill the blank of a Unity + WeChat applet AR development framework in the prior art and solve the technical problems that the existing applet AR development depends on a WeChat native tool, the model compatibility is limited, the research and development cost is high, and Unity resources cannot be reused.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer application technology, augmented reality (AR) technology and applet development technology, in particular to a WeChat applet AR development framework based on Unity and application thereof. BACKGROUND

[0002] With the popularization of augmented reality (AR) technology, WeChat applets, as high-frequency and low-threshold application carriers, have become an important platform for the landing of AR scenes. Currently, AR development based on WeChat applets mainly relies on native development using WeChat developer tools, which requires the establishment of a professional applet development team, resulting in a prolonged development cycle and a significant increase in manpower and time costs.

[0003] At the same time, existing applet AR applications generally rely on WeChat native AR plugins to provide spatial positioning and virtual-real fusion capabilities, but WeChat native AR plugins have strict restrictions on mobile hardware configurations (such as processor models and sensor accuracy), supporting only a portion of mainstream models and failing to cover mid- and low-end smartphone terminals, thereby limiting the user reach of AR applications.

[0004] In addition, although Unity engine is a mature three-dimensional content development tool with a large developer ecosystem and rich resource editing capabilities, there is currently a lack of standardized frameworks for seamlessly integrating AR content developed using Unity into WeChat applets, making it impossible to reuse the technical accumulation and resources of existing Unity development teams, further exacerbating the barriers and costs of applet AR development.

[0005] In summary, the existing technology has the core pain points of high development cost, poor model compatibility and inability to reuse Unity resources, and there is an urgent need for an AR development framework that can bridge the Unity and WeChat applet ecosystems, break through model restrictions and reduce development costs. SUMMARY

[0006] The present application provides a WeChat applet AR development framework based on Unity and application thereof, aiming to fill the gap in the existing technology of "Unity + WeChat applet" AR development frameworks and solve the technical problems of existing applet AR development relying on WeChat native tools, limited model compatibility, high development cost and inability to reuse Unity resources.

[0007] To achieve the above-mentioned purposes, the present application adopts the following technical solutions: The WeChat applet AR development framework based on Unity adopts a four-layer architecture of "applet side-Unity runtime-algorithm layer-resource layer", including seven core modules, which realize data intercommunication and collaborative work through standardized interfaces, as follows: The mini-program development portal entry module is used to receive user operations, complete user authentication and generate session tokens, load basic configuration data, realize bidirectional data transmission between the mini-program and Unity through a standardized message bridging mechanism, transmit session tokens and basic configuration data to the Unity embedding and runtime container module, and transmit user operation instructions to the mini-program and Unity hybrid rendering interaction module. The Unity embedding and runtime container module is used to load Unity WebGL format build artifacts based on basic configuration data, initialize the runtime environment and provide lifecycle management, connect to the terminal camera and IMU sensor to acquire camera image frames and IMU data streams, transmit the camera image frames and IMU data streams to the self-developed SLAM and 6DoF tracking algorithm library module, and report the runtime status to the mini-program development portal entry module. The self-developed SLAM and 6DoF tracking algorithm library module is used to receive camera image frames and IMU data streams, and perform feature extraction and matching, pose estimation, map update and tracking quality assessment through visual-inertial fusion technology. It outputs the camera pose data in the world coordinate system, tracking quality assessment results and sparse 3D map data to the coordinate system alignment and AR space management module. The coordinate system alignment and AR space management module is used to receive pose data, track quality assessment results and sparse 3D map data, define a unified Unity world coordinate system standard and convert camera coordinate system pose to Unity world coordinate system pose, detect the real plane and generate anchor point data, transmit the Unity world coordinate system pose to the terminal camera and virtual resource rendering fusion module, and transmit the anchor point data to the virtual resource editing, loading and management module. The virtual resource editing, loading, and management module is used to load virtual resource instances based on the resource server address in the basic configuration data, using a modular and layered strategy, bind anchor point data, and transmit the virtual resource instances to the terminal camera and virtual resource rendering fusion module. The terminal camera and virtual resource rendering fusion module is used to receive camera image frames, Unity world coordinate system poses and virtual resource instances. It uses dual rendering channels to fuse the camera image frames as the background layer and the virtual resource instances as the foreground layer for rendering, and outputs the composite image to the mini-program and Unity hybrid rendering interaction module. The Mini Program and Unity hybrid rendering interaction module is used to receive composite images and user operation commands, display the composite images in conjunction with the Mini Program's native UI, forward user operation commands to the Unity scene script and receive operation result feedback, and send the feedback information back to the Mini Program development portal entry module to form an interactive closed loop.

[0008] In this specification, the self-developed SLAM and 6DoF tracking algorithm library module does not rely on the WeChat native AR plugin. The pose estimation includes IMU pre-integration, reprojection error construction, and nonlinear least squares optimization steps. The output pose data is used to ensure the accuracy of coordinate system alignment, and the tracking quality evaluation result is used to determine whether to trigger the relocalization process.

[0009] In this specification, the coordinate system alignment and the Unity world coordinate system standard defined by the AR space management module are as follows: the world origin is the position of the camera pose in the first frame, the upward direction is the direction opposite to the direction of the phone's gravitational acceleration, and 1 unit corresponds to a scale of 1 meter. The coordinate system transformation is achieved through a linear transformation matrix to ensure the consistency between virtual resource instances and real world space.

[0010] In this specification, the modular layering strategy of the virtual resource editing, loading and management module is specifically to divide virtual resources into a basic model layer, a decoration effect layer and an interaction logic layer, package and export them through a custom Bundle format, load dependent resources at runtime using an incremental update method based on the resource version comparison results, and initialize virtual resource instances according to loading priority.

[0011] In this specification, the dual rendering channels of the terminal camera and virtual resource rendering fusion module are as follows: the first rendering channel renders camera image frames and adjusts exposure and white balance parameters; the second rendering channel renders virtual resource instances and supports shadow casting and ambient light occlusion effects, and optionally performs depth occlusion processing, achieving natural occlusion display by comparing the depth values ​​of virtual resource instances with those of the real scene.

[0012] In this specification, the standardized message bridging mechanism of the mini-program development portal entry module adopts a structured format of "message type + data content + timestamp + session token". User operation commands are transmitted in real time, Unity status feedback is returned at a frequency synchronized with the rendering frame rate, and the session token is used for full authentication.

[0013] In this specification, the IMU pre-integration step is based on the acceleration and angular velocity data in the IMU data stream, combined with the pose data of the previous moment to predict position, velocity and attitude. The prediction results are used for reprojection error construction and nonlinear least squares optimization to improve the stability of pose estimation.

[0014] In this specification, the coordinate system alignment and AR space management module uses the RANSAC algorithm to detect the real plane. It determines the optimal plane by randomly selecting three non-collinear 3D points to fit the initial plane and counting the number of interior points. The plane type is determined based on the plane normal vector. The generated anchor point data includes a unique anchor point ID and the position and orientation in the Unity world coordinate system.

[0015] In this specification, the virtual resource editing, loading, and management module supports resource caching management. Loaded virtual resource instances are cached locally and loaded directly upon the next startup. When resources are no longer in use, they are automatically unloaded to release memory. When resources are updated, only the resource package difference files are downloaded.

[0016] Applications of the Unity-based WeChat Mini Program AR development framework: Using any of the above-mentioned development frameworks, the development and implementation of AR applications for cultural interaction, scene roaming, and product display can be achieved. The application process includes a complete closed loop of user authentication, runtime initialization, data acquisition, pose estimation, coordinate alignment, resource loading, virtual-real fusion, interactive response, and resource recycling.

[0017] In summary, the present invention has at least the following beneficial effects: Significantly improved cross-device compatibility: Without relying on WeChat's native AR plugin, it only requires the terminal to support the camera and IMU sensor to achieve full-scenario smartphone terminal adaptation, greatly expanding the user reach of AR applications.

[0018] Reduced R&D costs and barriers to entry: By leveraging the existing Unity development team's technical expertise and resources, only a small number of mini-program integration personnel are needed. There is no need to build a separate mini-program development team, simplifying the development process and shortening the R&D cycle.

[0019] AR interactive experience optimization: The self-developed visual-inertial fusion SLAM algorithm achieves high-precision spatial positioning and stable tracking, ensuring the consistency of the position of virtual objects in real space; the virtual-real fusion rendering technology realizes the natural integration of real environment and virtual resources, enhancing immersion.

[0020] Improved resource management efficiency: Modular resource layering editing, dynamic loading and incremental update mechanism, adapt to different network environments and device performance, reduce resource download traffic and loading latency, and ensure smooth application operation.

[0021] The framework is highly scalable: standardized module interfaces and flexible architecture design support the rapid development and iteration of different types of AR scenarios (cultural interaction, scene roaming, product display, etc.) and adapt to diverse business needs. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the modules of the Unity-based WeChat Mini Program AR development framework involved in this invention.

[0023] Figure 2 This is a schematic diagram of the architecture of the Unity-based WeChat Mini Program AR development framework involved in this invention.

[0024] Figure 3This is a schematic diagram of the "Datang Everbright City" AR poetry interactive application involved in this invention.

[0025] Figure 4 This is a schematic diagram of the "Xia Ke Xing AR Roaming" application involved in this invention. Detailed Implementation

[0026] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0027] like Figure 1 and Figure 2 As shown, this embodiment provides a WeChat Mini Program AR development framework based on Unity, adopting a four-layer architecture of "Mini Program client - Unity runtime - algorithm layer - resource layer", including seven core modules. Each module achieves data interoperability and collaborative work through standardized interfaces, as detailed below: The mini-program development portal entry module is used to receive user operations, complete user authentication and generate session tokens, load basic configuration data, realize bidirectional data transmission between the mini-program and Unity through a standardized message bridging mechanism, transmit session tokens and basic configuration data to the Unity embedding and runtime container module, and transmit user operation instructions to the mini-program and Unity hybrid rendering interaction module. The Unity embedding and runtime container module is used to load Unity WebGL format build artifacts based on basic configuration data, initialize the runtime environment and provide lifecycle management, connect to the terminal camera and IMU sensor to acquire camera image frames and IMU data streams, transmit the camera image frames and IMU data streams to the self-developed SLAM and 6DoF tracking algorithm library module, and report the runtime status to the mini-program development portal entry module. The self-developed SLAM and 6DoF tracking algorithm library module is used to receive camera image frames and IMU data streams, and perform feature extraction and matching, pose estimation, map update and tracking quality assessment through visual-inertial fusion technology. It outputs the camera pose data in the world coordinate system, tracking quality assessment results and sparse 3D map data to the coordinate system alignment and AR space management module. The coordinate system alignment and AR space management module is used to receive pose data, track quality assessment results and sparse 3D map data, define a unified Unity world coordinate system standard and convert camera coordinate system pose to Unity world coordinate system pose, detect the real plane and generate anchor point data, transmit the Unity world coordinate system pose to the terminal camera and virtual resource rendering fusion module, and transmit the anchor point data to the virtual resource editing, loading and management module. The virtual resource editing, loading, and management module is used to load virtual resource instances based on the resource server address in the basic configuration data, using a modular and layered strategy, bind anchor point data, and transmit the virtual resource instances to the terminal camera and virtual resource rendering fusion module. The terminal camera and virtual resource rendering fusion module is used to receive camera image frames, Unity world coordinate system poses and virtual resource instances. It uses dual rendering channels to fuse the camera image frames as the background layer and the virtual resource instances as the foreground layer for rendering, and outputs the composite image to the mini-program and Unity hybrid rendering interaction module. The Mini Program and Unity hybrid rendering interaction module is used to receive composite images and user operation commands, display the composite images in conjunction with the Mini Program's native UI, forward user operation commands to the Unity scene script and receive operation result feedback, and send the feedback information back to the Mini Program development portal entry module to form an interactive closed loop.

[0028] In some embodiments, the self-developed SLAM and 6DoF tracking algorithm library module does not rely on the WeChat native AR plugin. The pose estimation includes IMU pre-integration, reprojection error construction, and nonlinear least squares optimization steps. The output pose data is used to ensure the accuracy of coordinate system alignment, and the tracking quality evaluation result is used to determine whether to trigger the relocalization process.

[0029] In some embodiments, the coordinate system alignment and the Unity world coordinate system standard defined by the AR space management module are as follows: the world origin is taken as the position of the camera pose in the first frame, the upward direction is taken as the direction opposite to the direction of the phone's gravitational acceleration, and 1 unit corresponds to a scale of 1 meter. The coordinate system transformation is achieved through a linear transformation matrix to ensure the consistency between the virtual resource instance and the real world space.

[0030] In some embodiments, the modular layering strategy of the virtual resource editing, loading and management module is to divide the virtual resources into a basic model layer, a decoration effect layer and an interaction logic layer, package and export them using a custom Bundle format, load dependent resources at runtime using an incremental update method based on the resource version comparison results, and initialize virtual resource instances according to loading priority.

[0031] In some embodiments, the dual rendering channels of the terminal camera and virtual resource rendering fusion module are specifically as follows: the first rendering channel renders camera image frames and adjusts exposure and white balance parameters; the second rendering channel renders virtual resource instances and supports shadow casting and ambient light occlusion effects; optionally, depth occlusion processing is performed, and natural occlusion display is achieved by comparing the depth values ​​of virtual resource instances with those of the real scene.

[0032] In some embodiments, the standardized message bridging mechanism of the mini-program development portal entry module adopts a structured format of "message type + data content + timestamp + session token". User operation instructions are transmitted in real time, Unity status feedback is returned at a frequency synchronized with the rendering frame rate, and the session token is used for full authentication.

[0033] In some embodiments, the IMU pre-integration step is based on the acceleration and angular velocity data in the IMU data stream, combined with the pose data of the previous moment, to predict position, velocity, and attitude. The prediction results are used for reprojection error construction and nonlinear least squares optimization to improve the stability of pose estimation.

[0034] In some embodiments, the coordinate system alignment and AR space management module uses the RANSAC algorithm to detect the real plane, fits the initial plane by randomly selecting three non-collinear 3D points, determines the optimal plane by counting the number of interior points, determines the plane type based on the plane normal vector, and the generated anchor point data includes a unique anchor point ID, position and pose in the Unity world coordinate system.

[0035] In some embodiments, the virtual resource editing, loading and management module supports resource caching management. Loaded virtual resource instances are cached locally and loaded directly upon the next startup. When resources are no longer in use, they are automatically unloaded to release memory. When resources are updated, only the resource package difference files are downloaded.

[0036] Applications of the Unity-based WeChat Mini Program AR development framework: Using any of the above-mentioned development frameworks, the development and implementation of AR applications for cultural interaction, scene roaming, and product display can be achieved. The application process includes a complete closed loop of user authentication, runtime initialization, data acquisition, pose estimation, coordinate alignment, resource loading, virtual-real fusion, interactive response, and resource recycling.

[0037] The technical concept of this invention is as follows: A Unity-based AR development framework for WeChat mini-programs and its application technology solutions I. Core Objectives of the Technical Solution This technical solution aims to address the current industry gap in the lack of a "Unity + WeChat Mini Program" AR development framework. It overcomes the technical pain points of existing WeChat Mini Program AR development, such as reliance on WeChat developer tools, the need for separate development teams leading to high R&D costs, and limitations imposed by WeChat's native AR plugin device models. By constructing a modular, highly compatible, and high-precision AR development framework, it achieves resource reuse of Unity development (requiring only a small number of additional Mini Program integration personnel), cross-smartphone terminal compatibility (supported by phones with sufficient performance), high-precision 6DoF (six degrees of freedom) tracking, and virtual-real fusion rendering, ultimately providing users with a stable, smooth, and immersive AR interactive experience.

[0038] This invention proposes an AR development framework with a four-layer architecture: "Mini Program - Unity Runtime - Algorithm Layer - Resource Layer". It includes seven core modules and achieves data interoperability and collaborative work through modular design and standardized interfaces.

[0039] The core of the framework lies in: using the "Unity Embedding and Runtime Container Module" as a bridge to embed UnityWebGL products into the WeChat Mini Program runtime environment, reusing Unity's 3D content development capabilities; using the "Self-developed SLAM and 6DoF Tracking Algorithm Library Module" as the core, achieving high-precision spatial positioning through visual-inertial fusion technology without relying on WeChat's native AR plugin; ensuring spatial consistency and visual coordination of virtual and real fusion through the "Coordinate System Alignment and AR Space Management Module" and the "Terminal Camera and Virtual Resource Rendering Fusion Module"; and achieving efficient resource management and adaptation through the "Virtual Resource Editing, Loading and Management Module".

[0040] The entire framework achieves end-to-end implementation of AR applications through a complete process of "user authentication - runtime initialization - data acquisition - pose estimation - coordinate alignment - resource loading - virtual-real fusion - interactive response - resource recycling". It supports the development of various AR scenarios such as cultural interaction and scene roaming, while taking into account compatibility, accuracy and development efficiency.

[0041] The core objectives include: First, to build a standardized AR development framework that enables seamless embedding and collaborative work between Unity engine products and WeChat mini-programs, reusing existing Unity development team resources and lowering the barrier to entry for mini-program AR development; second, to overcome the device limitations of WeChat's native AR plugins through self-developed core algorithms, achieving cross-smartphone compatibility and adaptation; third, to improve the spatial positioning accuracy and virtual-real fusion stability of AR applications, providing users with an immersive AR interactive experience; and fourth, to establish an efficient virtual resource management mechanism that enables modular editing, dynamic loading, and incremental updates of resources, adapting to different network environments and device performance.

[0042] II. Core Technology Architecture Design This framework adopts a four-layer architecture of "Mini Program client - Unity runtime - algorithm layer - resource layer", which includes seven core modules. Each module achieves data communication and collaborative work through standardized interfaces to ensure closed-loop data transmission and low latency (controlled within 100ms).

[0043] Architecture hierarchy 1. Mini Program: As the user entry point and global control center, it only contains the "Mini Program Development Portal Entry Module", which is responsible for user authentication, configuration loading, and message forwarding. It is the core hub for data interaction between the Mini Program and the Unity side. 2. Unity Runtime Layer: As the core execution carrier, it includes the "Unity Embedding and Runtime Container Module", "Terminal Camera and Virtual Resource Rendering Fusion Module", "Coordinate System Alignment and AR Space Management Module", "Virtual Resource Editing, Loading and Management Module", and "Mini Program and Unity Hybrid Rendering Interaction Module". It receives data from the mini program and is responsible for setting up the runtime environment, hardware data docking, rendering execution and interactive response. 3. Algorithm Layer: As the core capability support, it includes the "self-developed SLAM and 6DoF tracking algorithm library module" and the "self-developed hybrid rendering algorithm module". It receives hardware data and configuration parameters from the Unity runtime layer, outputs pose data and rendering control parameters, and provides the core technology for virtual-real fusion. 4. Resource Layer: As content support, it includes cloud resource servers and local resource caches, stores virtual resource packages (models, animations, effects, etc.), receives resource requests from the Unity runtime layer, and provides data support.

[0044] III. Technical Details of Core Modules (I) Mini Program Development Portal Entry Module 1. Module Function Positioning As the user entry point and global control hub for the entire AR application, it is responsible for user authentication, basic configuration loading, page routing management, network request forwarding, and message bridging with the Unity runtime, ensuring the consistency and security of data transmission between the mini-program and the Unity side.

[0045] 2. Input and Output Input: User actions (scanning QR code / searching to enter, button clicks, parameter adjustments), results returned by the WeChat Open Platform authentication interface, and basic configurations distributed by the cloud configuration server (including AR scene list, resource server address, and algorithm parameter thresholds). Output: User authentication result (session token), basic configuration data, page routing instructions, user operation instructions forwarded to Unity, and network request parameters; among which, the session token and basic configuration data are directly passed to the "Unity Embedding and Runtime Container Module", and the user operation instructions are passed to the "Mini Program and Unity Hybrid Rendering Interaction Module".

[0046] 3. Implementation of Key Technologies 3.1 User Authentication Process After a user enters the entry page via WeChat by "searching for the mini-program name" or "scanning the scene QR code", the module first calls the login interface provided by the WeChat Open Platform to obtain the user's temporary login credentials. Then, it sends the temporary login credentials, the mini-program's unique identifier, and the mini-program key to the developer's backend authentication server via HTTPS protocol. The authentication server verifies the legality of the temporary login credentials through the WeChat Open Platform interface and returns the user's unique identifier and session key. The module generates a user session token based on the user's unique identifier and stores it in the WeChat mini-program's local cache for identity verification of subsequent resource access and interactive operations. Finally, the authentication result is fed back to the front end: if successful, it triggers the basic configuration loading process; if it fails, it displays the message "Login failed, please try again".

[0047] 3.2 Loading Basic Configuration The basic configuration is structured data, obtained from a cloud configuration server via HTTPS, and contains the core parameters required by all subsequent modules. An example is shown below: The scene list contains at least two types of scenes. Each scene type includes a scene identifier, scene name, Unity package access address, resource server address, and algorithm threshold parameter (minimum number of feature points). Maximum reprojection error The network configuration includes request timeout and retries; the UI configuration includes navigation bar title and button styles. This configuration data is directly passed to the "Unity Embedding and Runtime Container Module" for Unity package loading, resource access, and algorithm parameter initialization, ensuring that subsequent modules do not need to obtain additional configuration information.

[0048] 3.3 Message Bridging Mechanism A standardized communication scheme of "message type + data content + timestamp + session token" is adopted, leveraging the communication capabilities of the WeChat Mini Program WebView component to achieve bidirectional data transmission between the Mini Program and Unity. The transmission protocol uniformly adopts a structured field combination, including four required fields, to ensure data integrity and traceability; the transmission frequency follows the rule of "real-time transmission triggered by user operation commands, and the status feedback transmission frequency synchronized with the Unity rendering frame rate (default 30Hz)".

[0049] The messages transmitted from the mini-program to Unity can be categorized into "user operation" and "configuration adjustment," with the data content varying depending on the message type. When the message type is "user operation," it includes the operation type (e.g., starting an animation, switching models, adjusting colors), target object information (e.g., animation name, model identifier), and operation parameters (e.g., animation speed, model scaling ratio). The timestamp is the millisecond-level time data of the message sending (denoted as ). The session token is a unique identifier generated for user authentication and is used for identity verification. This message is directly passed to the "Mini Program and Unity Hybrid Rendering Interaction Module," triggering the corresponding operation in the Unity scene.

[0050] In messages transmitted from Unity to the mini-program, message types can be categorized as "status feedback" and "result notification," etc. The data content includes target object information (such as animation name, model identifier), current status (such as playing, loaded, adjustment complete), progress data (such as animation playback progress percentage, resource loading progress), and a timestamp (denoted as...). The session token field is consistent with the message sent by the mini-program for data traceability and identity verification. This message is used by the mini-program to update the UI display, provide feedback to the user on the current operation result, and form an interactive loop.

[0051] (II) Unity Embedded and Runtime Container Module 1. Module Function Positioning It enables seamless embedding and lifecycle management of Unity engine products in the WeChat Mini Program environment, connects to terminal hardware capabilities (camera, IMU sensor), provides a basic operating environment for algorithm calculation and rendering execution, and receives output data (basic configuration, session token) from the "Mini Program Development Portal Entry Module". It provides hardware data and runtime support for the subsequent "Self-developed SLAM and 6DoF Tracking Algorithm Library Module" and "Terminal Camera and Virtual Resource Rendering Fusion Module".

[0052] 2. Input and Output Input: Basic configuration issued by the mini-program, user session token, Unity WebGL format build artifacts, and terminal hardware device capability data (camera permission status, IMU sensor support status); Output: Unity runtime initialization state, hardware device data (camera image frames, IMU data stream), and lifecycle event feedback (initialization complete / pause / resume / destroy); among which, the hardware device data is directly passed to the "self-developed SLAM and 6DoF tracking algorithm library module", and the initialization state and lifecycle feedback are sent back to the "mini-program development portal entry module" to ensure a closed loop of data flow.

[0053] 3. Implementation of Key Technologies 3.1 Unity Embedding Strategy In the Unity editor, select the "WebGL" platform, set the rendering path to "Forward Rendering," and the texture compression format to "ETC2" to ensure compatibility with mainstream mobile GPUs. Remove redundant modules from the Unity engine (such as 2D physics and network modules), retaining only the core modules for graphics rendering, the input system, and resource management, compressing the build output size to under 50MB to reduce loading latency. The loading order strictly follows the process: "Mini Program WebView loads the Unity WebGL launcher page → Launcher loads the Unity engine core library (script files, WebAssembly files) → Initializes the Unity scene manager and resource loader → Sends an 'initialization complete' signal to the mini program," ensuring the runtime environment is set up in an orderly manner. For performance optimization, set the WebGL memory limit to 2GB, enable incremental garbage collection, and lock the rendering frame rate to 30Hz (denoted as 30Hz). ), balancing performance and device power consumption.

[0054] 3.2 Lifecycle Management Interface It provides four standardized interfaces: Initialize (Init), Pause, Resume, and Destroy, strictly aligned with the lifecycle of the mini-program page to ensure state consistency. The Initialize interface is called when the mini-program page loads, its core operations being loading the engine library, initializing the rendering pipeline, and creating an algorithm library instance, outputting "Initialization successful" and related status information. The Pause interface is called when the mini-program page is hidden, its core operations being pausing the rendering pipeline, stopping sensor data acquisition, and saving the current scene state, outputting "Pause successful" and a scene state indicator. The Resume interface is called when the mini-program page is displayed, its core operations being resuming the rendering pipeline, restarting sensor data acquisition, and restoring the scene state, outputting "Resume successful" and a scene state indicator. The Destroy interface is called when the mini-program page is unloaded, its core operations being releasing GPU memory, destroying the algorithm instance, and clearing the memory cache, outputting "Destroy successful" and the amount of memory released (denoted as ). (Unit: MB)

[0055] 3.3 Hardware capability integration Camera permissions are granted via the WeChat Mini Program permission request interface. Once permissions are granted, a camera instance is created, and the resolution is set to 1280×720 and the frame rate to 30fps (the same as the rendering frame rate). (Consistent), the camera video stream is mapped to a Unity texture, which is directly used as the background input of the "Terminal Camera and Virtual Resource Rendering Fusion Module"; the IMU sensor interface is implemented through the mini-program sensor listening interface to acquire accelerometer and gyroscope data, with the sampling frequency set to 100Hz (denoted as ). The data includes accelerometer X-axis, Y-axis, and Z-axis data (denoted as...). , , (Unit: meters per square second), gyroscope X-axis, Y-axis, and Z-axis data (denoted as...) , , (unit: radians / second), and millisecond-level timestamps (denoted as ). This data stream is pushed in real time to the "self-developed SLAM and 6DoF tracking algorithm library module" via a message bridging mechanism for pose estimation, ensuring that the algorithm module does not need to acquire additional sensor data.

[0056] (III) Self-developed SLAM and 6DoF tracking algorithm library module 1. Module Function Positioning As the core algorithm support of the framework, it achieves real-time estimation of six degrees of freedom (position + attitude) of the terminal in three-dimensional space through visual-inertial fusion SLAM technology, providing a high-precision spatial positioning foundation for virtual-real fusion, without relying on WeChat's native AR plugin, thus overcoming device limitations. It accepts hardware data input (camera image sequences, IMU data streams) and algorithm threshold parameters in the basic configuration of the "Unity Embedding and Runtime Container Module," and outputs the camera's pose in the world coordinate system, tracking quality evaluation results, and sparse 3D map data, providing core input data for the "Coordinate System Alignment and AR Space Management Module."

[0057] 2. Input and Output Input: A sequence of camera images transmitted by the Unity Embedded module (denoted as...) ,in (Frame number, used to distinguish image frames at different times), IMU sensor data stream (acceleration denoted as...) , Frame number, three-dimensional vector, unit: meters per square second; angular velocity denoted as... 3D vector, unit: radians / second), camera intrinsic parameter matrix (denoted as...) A 3×3 matrix describing the geometric relationship of camera imaging, with elements of... , The algorithm threshold parameter (minimum number of feature points) configured in the mini-program. Maximum reprojection error ); Output: The pose of the camera in the world coordinate system (denoted as...) , Indicates the world coordinate system. Indicates the camera coordinate system. The frame number contains the rotation matrix. With translation vector Tracking quality assessment results (number of available feature points) Average reprojection error IMU residuals ), sparse 3D map data (including keyframes) Spatial feature points The keyframe number. (The feature point number is used for the feature point number). The pose data and tracking quality assessment results are directly input into the "coordinate system alignment and AR space management module", and the sparse 3D map data is used for its own map update and relocation.

[0058] 3. Algorithm Model Construction, Training, and Application 3.1 Algorithm Model Construction This module adopts a "visual-inertial fusion SLAM" model, which includes four sub-models: feature extraction, IMU pre-integration, pose optimization, and map update. The model state vector is defined as follows: ; in: for The position of the camera in the world coordinate system at any given time, a three-dimensional vector, in meters; for The velocity of the camera in the world coordinate system at any given time, a three-dimensional vector, in meters per second; for The attitude of the camera in the world coordinate system at any given time is a quaternion, without units, used to describe the camera's rotation state (pitch about the X-axis, yaw about the Y-axis, roll about the Z-axis). This represents the gravitational acceleration in the world coordinate system, a three-dimensional vector, with units of meters per square second. The default value is... ; for The accelerometer zero bias is a three-dimensional vector, with units of meters per square second, used to correct accelerometer measurement errors. for The zero bias of the gyroscope at any given time is a three-dimensional vector, with units of radians per second, used to correct gyroscope measurement errors.

[0059] 3.2 Model Training Process The training dataset uses the publicly available EuRoC MAV dataset, which contains 11 sequences: MH_01~MH_05, V1_01~V1_03, and V2_01~V2_03. The dataset covers various indoor and outdoor scenes and includes synchronized camera images, IMU data, and ground truth pose labels (denoted as MH_01~MH_05, V1_01~V1_03, and V2_01~V2_03). This ensures the diversity and accuracy of the training data.

[0060] Data preprocessing stage: processing camera images The image features are enhanced by performing grayscale conversion, Gaussian blur (with a 5×5 kernel size, achieving the blur effect through a weighted average of neighboring pixels), and histogram equalization. Sliding window mean filtering (with a window size of 5, taking the average of 5 consecutive frames) is used on the IMU data to remove outliers and ensure data stability.

[0061] The model parameters are trained using gradient descent, with the weighted sum of "reprojection error + IMU residual" as the loss function. Camera intrinsic parameters in iterative optimization model Gravitational acceleration Sensor zero bias initial value , Adjust the parameters until the loss function value converges to a preset threshold (less than 0.1). After training, save the optimal parameters for subsequent algorithm applications. The loss function formula is: ; in The total number of frames in the training data. for Frame reprojection error, This is the weight coefficient during the training phase (empirical value 0.1). for The IMU residual of the frame.

[0062] 3.3 Model Application Process The first step is feature extraction and matching: in the current camera image frame Extract the set of corner feature points ( , (The number of feature points in the current frame), by calculating the SIFT descriptor of the feature points (denoted as ). ), and the previous keyframe feature point set ( Establish a correspondence between the previous keyframe number and the keyframe sequence number. This ensures the accuracy of feature point matching between frames.

[0063] The second step is IMU pre-integration: based on IMU data, the pose of the previous time step is predicted, including predictions of three dimensions: position, velocity, and attitude, with time intervals. Seconds (calculated from the rendering frame rate): Location prediction formula: ; in for Predicted camera position at any given time. for The real position of the camera in the world coordinate system at any given time (output result of the previous frame). for camera speed at all times for The rotation matrix corresponding to the camera pose at any given time (composed of quaternions) (Transformed) for Accelerometer readings at all times for The accelerometer zero bias is calculated at any given time. The calculation process is as follows: first, calculate the acceleration correction value (subtract the zero bias), transform it to the world coordinate system through a rotation matrix, subtract the gravitational acceleration to obtain the effective acceleration, then perform a second integration over time, and add the initial position and velocity terms to obtain the position prediction value.

[0064] Speed ​​prediction formula: ; in for The predicted value of the camera velocity at any given time is given, and the definitions of other parameters are consistent with the position prediction formula. The calculation process is as follows: multiply the effective acceleration by the time interval, add the velocity of the previous time step, and obtain the predicted velocity value.

[0065] Attitude prediction: via gyroscope angular velocity (Subtract zero bias) Performing quaternion integration, we obtain... Predicted values ​​of camera pose at any given time The integration process uses discretization to convert continuous angular velocity changes into discrete quaternion updates, with the update formula being: ,in This is quaternion multiplication. This is the rotational quaternion corresponding to the angular velocity.

[0066] The third step is to construct the visual measurement model and reprojection error: This involves converting the 3D points in the sparse 3D map... Through camera intrinsic parameter matrix With predicted pose (Depend on and (Components) Projected onto the current image frame The projection formula for the image plane is: ; in For three-dimensional points The projected pixel coordinates (two-dimensional vectors) are calculated as follows: First, the 3D points are transformed from the world coordinate system to the camera coordinate system (using rotation and translation vectors), and then converted to pixel coordinates using the camera intrinsic parameter matrix. Subsequently, the observed pixel points are calculated. (feature points) (actual pixel coordinates) and projection point reprojection error (Two-dimensional vector, unit: pixels).

[0067] The fourth step is pose optimization and fusion: An error function is constructed by weighted summing of the reprojection error and the IMU residual. The error function formula is as follows: ; in for Total frame error Let L2 be the square of the single reprojection error. This is the weighting coefficient for the application phase (empirical value 0.1, balancing visual error and IMU error). For IMU residuals (from location prediction error) Speed ​​prediction error Attitude prediction error (Composition). The Levenberg-Marquardt nonlinear least squares algorithm is used to iteratively optimize the error function and adjust the pose parameters. until the total error The optimal pose is obtained when the value converges to the minimum (less than 0.05). .

[0068] The fifth step is map updating and keyframe selection: calculating the disparity between the current frame and the previous keyframe. (The average deviation of the projected position of the feature point) If parallax Exceeding the threshold (empirical value of 5 pixels) or a decrease in tracking quality (number of feature points) near If the current frame is set as the new keyframe, then the current frame will be set as the new keyframe. Triangulation of new feature points (calculating 3D spatial coordinates by matching the poses and feature points of two keyframes). The map is sparsely mapped and a keyframe culling strategy (removing redundant keyframes and retaining the keyframes with the most information) and local bundle adjustment (optimizing the poses of local keyframes and 3D points) are used to maintain the map size and accuracy.

[0069] The sixth step is to track quality assessment and anomaly handling: calculate the number of available feature points. Average reprojection error IMU residuals (The average value of the total IMU error); if or If the tracking fails, it is marked as low-quality tracking, triggering a relocalization process (by matching the feature points of the current frame with the features of all key frames in the map to re-estimate the pose). If the relocalization fails, the mini-program is notified that "tracking failed, please adjust the environment".

[0070] (iv) Coordinate system alignment and AR spatial management module 1. Module Function Positioning Receives the optimal pose output by the "self-developed SLAM and 6DoF tracking algorithm library module". Based on the tracking quality assessment results, the camera coordinate system pose is uniformly converted to the Unity world coordinate system, providing plane detection and anchor point management functions to ensure that the position and orientation of virtual objects in space are aligned with the real world, providing aligned pose data and anchor point information for the "terminal camera and virtual resource rendering fusion module".

[0071] 2. Input and Output Input: Camera pose output by the self-developed 6DoF algorithm module Tracking quality assessment results, defining parameters for the Unity world coordinate system (world origin). Upward ,scale (1 unit corresponds to 1 meter). Output: Camera pose in Unity world coordinate system ( Using Unity's world coordinate system, Using the original world coordinate system, (Frame number), detected plane information (plane type: horizontal / vertical, plane parameter: normal vector) ,distance ), anchor data (anchor ID) The position of the anchor point in the Unity world coordinate system with posture The aligned pose data is passed to the "Terminal Camera and Virtual Resource Rendering Fusion Module", and the anchor point data is passed to the "Virtual Resource Editing, Loading and Management Module" for the placement and positioning of virtual objects.

[0072] 3. Implementation of Key Technologies 3.1 Coordinate System Transformation and Unification A coordinate system alignment strategy is adopted to define a unified Unity world origin. (Position based on the camera pose in the first frame) (as the origin of the world) and upward direction (Opposite to the direction of the phone's gravitational acceleration, i.e., perpendicular to the ground and upwards), scale (1 unit corresponds to 1 meter, consistent with the real-world scale). The transformation process from the camera coordinate system to the Unity world coordinate system is as follows: first, the camera pose is... rotation matrix With translation vector Perform coordinate transformation (adjust the coordinate axis orientation so that the axis orientation of the Unity camera coordinate system is consistent with that of the world coordinate system; the transformation matrix is ​​denoted as...). (a 3×3 matrix), then scaled. Obtain the camera pose in the Unity world coordinate system. The conversion process strictly follows the linear transformation rules to ensure the accuracy of the pose data.

[0073] 3.2 Plane Detection Algorithm Based on aligned point cloud data (from 3D points in a sparse 3D map) (Converted to Unity world coordinate system) Perform plane detection using the RANSAC algorithm (Random Sample Consensus Algorithm): randomly select 3 non-collinear 3D points. , , Fit an initial plane, the plane equation is: ( (For any point on the plane); calculate the distance from other 3D points to the plane. The statistical distance is less than the threshold (empirical value 0.05 meters, denoted as ). Number of interior points Repeat this process 100 times, selecting the plane with the most interior points as the optimal plane; based on the plane's normal vector... Determine the plane type (normal vector and direction of gravitational acceleration) An angle less than 30° is considered a horizontal plane, and an angle greater than 60° is considered a vertical plane. Output plane parameters (normal vector). ,distance ).

[0074] 3.3 Anchor Point Management When a user clicks through the mini-program's UI, the mini-program will record the pixel coordinates of the clicked location. This module receives the information via a message bridge mechanism. The camera pose is based on the Unity world coordinate system. With internal reference Convert pixel coordinates to rays in the world coordinate system (the ray originates at the camera position). ray direction After rotation matrix (Obtained through conversion); the intersection of the ray and the detected plane is the location of the anchor point. The attitude of the anchor point With plane normal vector Consistent (horizontal plane anchor point attitude is the default attitude) The orientation of the vertical plane anchor point adjusts with the plane normal vector. , (The rotation matrix corresponding to the normal vector); assign a unique ID to each anchor point. (In the format "Anchor_XXX", where XXX is an auto-incrementing number), set the anchor point position. ,attitude ID The data is bound to anchor points and stored in the anchor point management list. At the same time, the anchor point data is passed to the "Virtual Resource Editing, Loading and Management Module" to ensure that the virtual object is bound to the anchor point and maintains a stable position in subsequent frames.

[0075] (v) Virtual Resource Editing, Loading and Management Module 1. Module Function Positioning Deployed on both the Unity editor and runtime sides, it receives anchor point data for the "Coordinate System Alignment and AR Space Management Module" and basic configuration (resource server address) for the "Mini Program Development Portal Entry Module". It supports developers in creating and editing virtual resources in Unity, and provides virtual resources (models, animations, effects, etc.) to be rendered for the "terminal camera and virtual resource rendering fusion module" through standardized packaging, dynamic loading, and version management.

[0076] 2. Input and Output Inputs: Resource editing operations on the Unity editor side (model creation, animation editing, effects configuration), runtime resource loading commands (user operations from the mini-program, such as switching models), and anchor point data. Resource server address Resource version information ; Output: Standardized resource package (Editor side) Loaded virtual resource instance (Runtime side, including model mesh) Material Animation controller Special effects components After the virtual resource instance is bound to anchor point data, it is passed to the "Terminal Camera and Virtual Resource Rendering Fusion Module".

[0077] 3. Implementation of Key Technologies 3.1 Resource Editing and Layering When editing resources in the Unity editor, developers use a modular, layered strategy, dividing resources into a base model layer, a decorative effects layer, and an interaction logic layer. The base model layer contains core models such as character models and architectural models, and includes basic meshes. With material The decorative effect layer includes particle effects. Lighting effects Fog effect Resources to enhance the visual experience; the interaction logic layer includes animation clips. Interactive scripts Resources that enable interactive functionality. Each layer of resources is independent and supports individual editing and replacement; during the editing process, a unique identifier is added to each resource. With resource description file (Including resource types) Dependence on resources Loading priority (Values ​​range from 1 to 5, with 1 being the highest) to ensure accurate identification and loading during runtime.

[0078] 3.2 Resource Packaging and Export After editing, use our self-developed resource packaging tool Resources are categorized and packaged according to scenario, using a custom Bundle format, and include resource data (model mesh). Texture Animation curves ), resource description file Version information (Version number) Update time The texture is compressed during the packaging process (using ETC2 format, compression ratio). ), simplify the mesh of the model (preserve key details, reduce the face count by a certain percentage). ), optimize resource package size; after packaging, the developer will... Upload to the resource server address specified in the basic configuration. To achieve unified management, the resource package storage path is... ( (For scene identification).

[0079] 3.3 Runtime Resource Loading and Management At runtime, the module loads resources according to the mini-program's resource loading instructions (such as "load the Tang Dynasty Never Sleeps scene resources", including scene identifiers). ), from the resource server address Get the version information of the corresponding resource package The version of the resource cached locally Compare: If there is no local cache or the versions are inconsistent ( If so, the resource package will be downloaded via the HTTPS protocol. (Supports resuming interrupted downloads; download progress is reported to the mini-program during the download process.) =Downloaded bytes / Total bytes x 100%; After downloading, unzip the resource package and perform an MD5 checksum verification (checksum value). Verify resource integrity; if verification fails, re-download; based on the resource description file. Loading dependency resources According to loading priority Initialize resource instances sequentially (basic model layer) Load first, then the decorative effect layer. Interaction Logic Layer ); will resource instance With anchor data Binding sets the position of a resource in the Unity world coordinate system. with posture Finally, the resource instance is passed to the "Terminal Camera and Virtual Resource Rendering Fusion Module".

[0080] Additionally, the module supports resource caching and incremental updates: loaded resources are cached to a specified local path. It will be loaded directly on the next startup; when resources are updated, only the incremental part (resource package difference file) will be downloaded. The resource instance is 10%-30% the size of the complete resource package, reducing download traffic and time; when the resource is no longer in use (such as when the user switches scenarios), the resource instance is automatically unloaded. Release memory (Memory usage).

[0081] (vi) Terminal camera and virtual resource rendering fusion module 1. Module Function Positioning Camera image textures that support the "Unity Embedded and Runtime Container Module" Unity world coordinate system pose in the "Coordinate System Alignment and AR Space Management Module" Virtual resource instances in the "Virtual Resource Editing, Loading and Management Module" It uses a self-developed hybrid rendering algorithm to achieve the fusion rendering of real camera images and virtual resources, and outputs the composite image to the mini program for display, providing users with an immersive AR experience.

[0082] 2. Input and Output Input: Camera image texture (Resolution 1280×720, RGBA format), Camera pose in Unity world coordinate system Virtual resource instances Rendering parameters (exposure) White balance Resolution (from basic configuration); Output: Composite image after blending and rendering (Texture format RGBA, resolution consistent with camera image), passed to the "Mini Program and Unity Hybrid Rendering Interaction Module", and finally mapped to the Mini Program canvas for display.

[0083] 3. Key Technology Implementation (Application of Self-Developed Hybrid Rendering Algorithm) 3.1 Rendering Coordinate Alignment Based on the camera pose in the Unity world coordinate system Setting camera parameters in the Unity rendering pipeline: View Matrix ( (A 3×3 identity matrix describing the camera's viewing angle in the world coordinate system), projection matrix. From the camera intrinsic parameter matrix With image resolution The calculation formula is as follows: ( (This is the horizontal field of view of the camera, calculated from intrinsic parameters) to ensure that the projection of virtual resources is consistent with the imaging effect of the real camera.

[0084] 3.2 Background Texture Mapping Camera image texture Bind to a fullscreen quad, render the quad in the first render pass (Pass 1) of the Unity render pipeline, in the bottom layer (render queue). This involves rendering a realistic world background; during rendering, the image exposure is adjusted based on the rendering parameters in the basic configuration. (Parameter range is 0.5~2.0, adjusted by the texture brightness value) White balance (By adjusting the gain of the RGB channels) , , This ensures that white objects remain white under different lighting conditions (gain value range 0.8~1.2), and guarantees uniform brightness and true color in the background image.

[0085] 3.3 Virtual Object Rendering Render virtual asset instances in the second rendering pass (Pass 2) of the Unity rendering pipeline. Rendering queue (Above the background layer): First, perform vertex transformation on the virtual resources (via the view matrix). With projection matrix Vertex coordinates of virtual resources Convert to screen coordinates Then, lighting calculations are performed (combining scene lighting parameters). Calculate the light intensity on the surface of the virtual resource. , The diffuse reflection coefficient of the material. For material specular highlight factor), material rendering (material properties applied to virtual resources). Based on the alignment results of the anchor points with world coordinates, determine the position of the virtual resource in three-dimensional space. Orientation Supports shadow casting (virtual resources casting shadows onto a real background, shadow map resolution). Ambient occlusion (enhances the three-dimensionality of virtual resources, occlusion intensity) Effects such as these enhance the realism.

[0086] 3.4 Depth and Occlusion Handling (Optional) Through depth estimation module Generate approximate depth map (Depth value based on texture gradient and feature point distance information of camera image) (Range 0-10 meters); when rendering virtual objects, compare the depth values ​​of the pixels on the surface of the virtual object. Depth value at the corresponding location in the depth map :like (Error threshold 0.1 meters), then this pixel will not be rendered (achieving virtual object occlusion in a real scene); if Then, normal rendering (implementing virtual objects occluding real scenes) is performed, thus handling some occlusion relationships.

[0087] 3.5 Final Composite Output Background layer (real camera image) ) and virtual object foreground layer (virtual resource rendering result) Perform color correction (to make the color of the virtual object consistent with the background color, with a color adjustment factor). ~1.05), Brightness Balance (adjusting the brightness of virtual objects) ~1.1, to avoid being too bright or too dark), Gamma correction (Gamma value) After adapting to the display characteristics of different devices, the data is synthesized; the synthesis formula is as follows: ,in Adjust the transparency of the virtual layer (default 0.95, adjustable via configuration); the composited image. It is a texture format with the same resolution as the camera image. It is passed to the "Mini Program and Unity Hybrid Rendering Interaction Module" and finally mapped to the Mini Program canvas for display.

[0088] (vii) Mini Program and Unity Hybrid Rendering Interaction Module 1. Module Function Positioning The composite image is generated by the "terminal camera and virtual resource rendering fusion module". The "Mini Program Development Portal Entry Module" provides user operation commands, while the "Unity Embedding and Runtime Container Module" provides status feedback, enabling collaborative display and interactive response between the Mini Program front-end UI and the Unity 3D rendering view, forming a complete user interaction loop.

[0089] 2. Input and Output Input: Composite image User operation instructions Unity status feedback ; Output: Collaborative display screen (Mini Program UI + AR rendering view), operation commands forwarded to Unity Status information fed back to the mini-program .

[0090] 3. Implementation of Key Technologies 3.1 Collaborative Display Layout The layout adopts a "top navigation bar + middle AR view + bottom operation bar": the top navigation bar is the native UI of the mini program and includes a title. Back button ,high The central section is an AR view rendered by Unity, occupying the main area of ​​the screen, with a width of... ,high ( For screen height, (Height of the bottom action bar), the composite screen is displayed in full screen using the WebView component. The bottom operation bar uses the native UI of the mini-program and includes function buttons (such as switching models). Startup animation ,Photograph ), button spacing The layout is achieved through the mini-program's flexible layout (flex layout), ensuring that the AR view and the native UI adapt to different screen sizes (supporting screen resolutions from 320×480 to 1080×2400).

[0091] 3.2 Operation Command Forwarding and Response When a user clicks a function button in the bottom operation bar, the mini-program will send the operation command. (Including operation type) Target parameters The command is passed to this module through a message bridging mechanism. This module performs format validation (verifies the validity of the session token and the integrity of the parameters) and then converts it into an operation command that Unity can recognize. And forward it to the Unity scene script; the Unity scene script adjusts the scene state according to the instructions (such as loading new virtual resource instances when switching models). The animation controller is triggered when the animation starts. ), and the operation results Feedback (such as model switching completion, animation playback status) is sent to this module; this module converts the feedback results into status information that the mini-program can recognize. The "Mini Program Development Portal Entry Module" is returned to update the UI display (e.g., displaying a "Model Switching Successful" message). Animation progress bar ).

[0092] 3.3 Photo taking and sharing function When the user clicks the photo button At that time, the mini-program sends a "screenshot command" to Unity through this module. Unity calls the rendering pipeline's screenshot interface to capture the current composite image. And send it back to the mini-program; the mini-program receives the screenshot. Then, overlay an event watermark (such as the scene name). Current time (or QR code) (Includes a link to enter the mini-program) The watermark is located in the bottom right corner of the image, and its size is... Through WeChat Open Platform API Save images to the user's album; also provide a sharing function, allowing users to share via the mini-program's native sharing interface. Take screenshot Link to the scene Share to WeChat friends or Moments, share title +"AR Interactive Experience", share description "Come and experience immersive AR scenes together!"

[0093] IV. Typical Application Process (Complete Closed Loop) 1. User entry into the mini program: Users enter the entry page by searching for the mini program name on WeChat or scanning the scene QR code. The mini program development portal entry module completes user authentication and basic configuration loading, and displays a list of scenes to the user (such as "Datang Everbright City AR Poetry Interaction" and "Xia Ke Xing AR Roaming"). 2. Unity runtime initialization: After the user selects the target scene, the mini-program calls the Unity Embedded and Runtime Container module to load the Unity WebGL package corresponding to the scene; the Unity runtime initializes the rendering pipeline, scene manager, and algorithm library instance, establishes a message channel with the mini-program, and sends a "initialization complete" message to the mini-program; 3. Camera and sensor data acquisition: The mini-program accesses camera permissions to obtain real-time camera image frames. Unity embedded module synchronously acquires IMU data stream (accelerometer) gyroscope The two types of data are fed into the self-developed SLAM and 6DoF tracking algorithm library module in real time. 4. Self-developed algorithm library for 6DoF tracking and map building: The algorithm library analyzes camera image frames... Perform feature extraction and matching, combine IMU data for pre-integration and pose optimization, and output the optimal camera pose. Simultaneously update the sparse 3D map with the tracking quality assessment results; 5. Coordinate System Alignment and AR Content Placement: The coordinate system alignment and AR space management module will adjust the camera pose... Convert to Unity world coordinate system pose Detect the real plane and generate anchor points The anchor point data is then passed to the virtual resource editing, loading, and management module. 6. Virtual Resource Loading and Hybrid Rendering: The virtual resource module loads resources from the resource server according to the scene configuration. Load virtual resource instance After binding the anchor point data, it is passed to the terminal camera and virtual resource rendering fusion module; the rendering fusion module will then combine the camera images... As a background, virtual resources are overlaid and occlusion relationships are handled to output a composite image. ; 7. Interaction and Business Logic Execution: Users can adjust the scene (such as starting the animation or switching the poem) through the mini-program's operation bar. The operation command is forwarded and triggers the Unity scene adjustment. The adjustment result is fed back to the mini-program in real time and updates the UI display. 8. Session End and Resource Reclamation: When the user clicks the back button or closes the mini-program, the mini-program notifies the Unity runtime container module to release resources (textures). Memory caching (This will close the camera and sensor permissions, clear local cache data, and complete the session loop.)

[0094] V. Examples Example 1: "The Tang Dynasty Never Sleeps" AR Poetry Interactive Application Developers created the "Tang Dynasty Never Sleeps" themed resources using the virtual resource editing, loading, and management module in the Unity editor: (Reference) Figure 3 The basic model layer contains a model of a little girl dressed in bright red Tang Dynasty-style clothing (model ID). Includes standby, waving sleeve, and spinning animation clips. ), pink clouds ( ), lantern buildings ( ),moon( Background models such as ) and decorative effect layers include falling petal particle effects ( number of particles (number per second), warm-toned lighting ( Color temperature ), fog effect ( ,concentration The interaction logic layer contains the title text "The Tang Dynasty Never Sleeps City" (). ), vertical classical Chinese poems ( to Includes highlighted animation ), Start button ( ), camera button ( ), Back button Interactive scripts Resources are packaged to generate a resource package. Upload to the resource server address .

[0095] After the user opens the mini-program and selects the scene, Unity runtime initialization is complete (initialization time). (seconds), the camera begins acquiring real-world images (1280×720 resolution), and the algorithm library estimates the camera pose in real time. (mean reprojection error) (pixels), coordinate system alignment module detects horizontal plane (number of interior points) And generate anchor points. The entire virtual scene of "Tang Dynasty Never Sleeps" is bound to an anchor point. , The virtual resource module loads the little girl model, background model, and poem resources (loading time). (seconds), the rendering and blending module uses the real environment as a background and overlays a virtual scene, adjusting the lighting (exposure) White balance , , Make the virtual resources consistent with the real environmental lighting style.

[0096] The user clicks the start button , and the mini-program sends a "start performance" instruction , and Unity controls the little girl model to play a dancing animation (animation speed ), and at the same time triggers the petal falling special effect and the lighting animation (the lighting brightness fades gradually with the animation progress, range 0.8 - 1.2); when the poem button is clicked , the mini-program plays the poem recitation audio (audio format MP3, sampling rate 44.1kHz), and Unity triggers the highlighting animation of the corresponding poem sentences, lighting the sentences one by one (highlight color , lighting duration seconds / sentence); when the photo-taking button is clicked , the mini-program captures the current synthesized screen, superimposes the "Grand Tang Mall" watermark and the QR code (QR code size 80px × 80px), saves it to the album and supports sharing. When the user exits the scene, Unity releases the resources (releases memory ), and the mini-program closes the camera permission and clears the cache.

[0097] Example 2: "Roaming in AR of The Song of the Gallant Swordsman" application The developer makes the resources themed on "The Song of the Gallant Swordsman" in the Unity editor: Referring to Figure 4 , the basic model layer includes the gallant model ( , including standby, cupping one hand in the other before the chest, and waving the sleeve animations), the poet model ( , including holding the zither and playing animations), the bamboo forest ( ), the pavilions and towers ( ), the water surface ( ), the lotus flowers ( ), the floating lanterns ( ), etc.; the decorative effect layer includes the water surface reflection effect ( , reflection intensity 0.7), the ripple special effect ( , ripple frequency 2Hz), the flickering effect of the lights ( , flickering frequency 1Hz); the interaction logic layer includes the title text of "The Song of the Gallant Swordsman" ( ), the vertical ancient poem sentences ( to , including fade-in and fade-out animations), the help button ( ), the rotate view button ( ), the enter AR / exit AR button ( / ). After the resources are packaged, a resource package is generated and uploaded to the resource server address .

[0098] Users launch the mini-program and click the "Enter AR" button. When the mini-program launches the camera, Unity loads scene resources at runtime (loading time) (seconds), the algorithm library collects camera and IMU data and estimates pose. (Tracking success rate 96%); The coordinate system alignment module uses the water surface as the primary bearing plane (normal vector). ,distance (meters), generate multiple anchor points: The pavilion is associated with the poet. Binding heroes to Attach floating lamps to the bamboo forest. When the user moves their phone, Unity adjusts the camera pose accordingly. The projection position of virtual resources is updated in real time (update frequency 30Hz) to achieve a "roaming" effect; the rendering and blending module handles the occlusion relationship between floating lamps and the real ground (depth threshold 0.1 meters) to enhance immersion.

[0099] The user clicks the rotate view button The mini-program sends the "adjust view" command. Unity adjusts camera pose This allows for a 360-degree view of the virtual scene; click the help button. The mini-program displays operation instructions (such as "roam the scene with your mobile phone" and "click on the poem to play the audio"); when exiting AR mode, the mini-program hides the camera view and displays the scene introduction page (including the scene background story and operation instructions), and the user can click "enter AR" again to return to the AR scene.

[0100] VI. Core Advantages and Protection Points Core advantages 1. Cross-device compatibility: Adopting self-developed SLAM and 6DoF tracking algorithms, it does not rely on WeChat's native AR plugin. As long as the mobile phone supports the camera and IMU sensor and meets the performance requirements (CPU main frequency ≥ 1.5GHz, memory ≥ 2GB), it can cover all smartphone terminals (Android 7.0 and above, iOS 11.0 and above), solving the problem of device limitations of existing technologies; 2. Low R&D costs: Reusing existing Unity development team resources, only 2-3 mini-program integration personnel are needed, eliminating the need to build a separate mini-program development team, shortening the development cycle by 50% and reducing R&D costs by 60%; 3. High precision and high stability: The visual-inertial fusion SLAM algorithm achieves millimeter-level pose estimation, with an average reprojection error of less than 2.0 pixels and a tracking success rate of over 95%, ensuring the stability and accuracy of the AR experience; 4. Highly efficient resource management: Adopting modular resource tiering, incremental updates, and caching mechanisms, the resource package size is reduced by 40%, download traffic is reduced by 70%, and it adapts to different network environments (2G / 3G / 4G / 5G / Wi-Fi) and device performance; 5. Smooth interactive experience: The mini-program and Unity collaborate in rendering, with operation command transmission latency of less than 100ms and a stable rendering frame rate of 30Hz. There are no stutters or jitters, resulting in a strong sense of immersion.

[0101] Desiring a protected point 1. Self-developed SLAM and 6DoF tracking algorithms: Achieve high-precision pose estimation through visual-inertial fusion, including a complete process of feature extraction and matching, IMU pre-integration, pose optimization, and map update, without relying on WeChat's native AR plugin, thus overcoming device limitations; 2. Embedding and Communication Mechanism between Unity and WeChat Mini Programs: Loading strategies (build configuration, loading order, performance tuning), lifecycle management interfaces (Init / Pause / Resume / Destroy), and bidirectional message bridging protocols (message format, transmission frequency, authentication) for embedding Unity WebGL artifacts into WeChat Mini Programs. 3. Multi-coordinate system alignment strategy: Transformation rules (unified world origin, up direction, scale definition) and linear transformation matrix construction method for camera coordinate system, Unity world coordinate system, and device coordinate system; 4. Virtual-Real Fusion Rendering Algorithm: Layered rendering process of camera background and virtual resources (dual rendering channels, rendering queue settings), depth occlusion handling methods (depth map estimation, pixel-level depth comparison), color and brightness correction strategies; 5. Modular management solution for virtual resources: layered resource editing (basic model layer / decorative effect layer / interaction logic layer), standardized packaging format, dynamic loading and incremental update mechanism (version comparison, difference file download, cache management).

[0102] 3DoF / 6DoF: 3DoF and 6DoF are terms describing the degrees of freedom of an object's motion in space, commonly used in virtual reality (VR) and augmented reality (AR) technologies. DoF is an abbreviation for Degrees of Freedom.

[0103] 3DoF (Three Degrees of Freedom): 3DoF indicates that an object has three degrees of freedom in space: rotational motion, including rotation around the X-axis (pitch), rotation around the Y-axis (yaw), and rotation around the Z-axis (roll). In this case, the object cannot move freely in space, but can only rotate in place. This is common in some early VR or AR devices, such as Google Cardboard.

[0104] 6DoF (Six Degrees of Freedom): 6DoF indicates that an object has six possible motions in space: in addition to the three rotational motions of 3DoF, it also includes translational motions along the X-axis (left / right), Y-axis (up / down), and Z-axis (back / forward). This means that the object can move and rotate freely in space. This is common in advanced VR or AR devices such as the Oculus Rift or HTC Vive.

Claims

1. A Unity-based WeChat applet AR development framework, characterized in that, The application comprises: a mini-program development portal entry module for receiving user operations, completing user authentication and basic configuration loading, realizing two-way data transmission between the mini-program and Unity, distributing session tokens, configuration data and user operation instructions; a Unity embedding and runtime container module for loading Unity WebGL build products and initializing the running environment, interfacing with terminal hardware to obtain camera images and IMU data, and providing life cycle management and state feedback; a self-developed SLAM and 6DoF tracking algorithm library module for processing camera images and IMU data, realizing six-degree-of-freedom pose estimation, map updating and tracking quality evaluation, and outputting pose data and related evaluation results; a coordinate system alignment and AR space management module for defining a Unity world coordinate system standard, converting camera poses and detecting real planes, and generating anchor point data to ensure virtual resource space alignment; a virtual resource editing, loading and management module for loading virtual resources according to a modularization strategy, binding anchor point data and outputting virtual resource instances to a rendering module; a terminal camera and virtual resource rendering fusion module for receiving camera images, pose data and virtual resource instances, performing virtual-real fusion rendering and outputting a synthesized picture; a mini-program and Unity hybrid rendering interaction module for displaying the synthesized picture and the mini-program native UI, forwarding operation instructions and feeding back interaction results, forming a closed loop.

2. The Unity-based WeChat Mini Program AR development framework of claim 1, wherein, The self-developed SLAM and 6DoF tracking algorithm library module does not rely on the WeChat native AR plug-in, and through visual-inertial fusion technology, it performs feature extraction and matching, IMU pre-integration, re-projection error construction and nonlinear least squares optimization to complete six-degree-of-freedom pose estimation, and the output tracking quality evaluation results are used to determine whether to trigger the repositioning process.

3. The Unity-based WeChat Mini Program AR development framework of claim 1, wherein, The coordinate system alignment and AR space management module defines the Unity world coordinate system standard as follows: taking the first frame camera pose position as the world origin, taking the direction opposite to the phone gravity acceleration direction as the upward direction, and taking 1 unit corresponding to 1 meter scale, and completing the conversion of the camera coordinate system to the pose Unity world coordinate system through a linear transformation matrix.

4. The Unity-based WeChat Mini Program AR development framework of claim 1, wherein, The modularization strategy of the virtual resource editing, loading and management module is as follows: the virtual resources are divided into a basic model layer, a decoration effect layer and an interaction logic layer, packaged and exported through a self-defined Bundle format, loaded in an incremental update manner based on resource version comparison results during runtime, and virtual resource instances are initialized according to priority.

5. The Unity-based WeChat Mini Program AR development framework of claim 1, wherein, The terminal camera and virtual resource rendering fusion module realizes virtual-real fusion through two rendering channels: the first rendering channel renders the camera images and dynamically adjusts the exposure and white balance parameters; the second rendering channel renders the virtual resource instances, supports shadow projection and ambient light shading effects, and performs depth occlusion processing to realize natural occlusion relationship.

6. The Unity-based WeChat Mini Program AR development framework of claim 1, wherein, The two-way data transmission of the mini-program development portal entry module adopts a standardized message bridging mechanism, the message format includes message type, data content, timestamp and session token, user operation instructions are transmitted in real time, Unity state feedback is transmitted back at a frequency synchronized with the rendering frame rate, and the session token is used for whole-process identity verification.

7. The Unity-based WeChat Mini Program AR development framework of claim 2, wherein, The IMU pre-integration step is based on acceleration and angular velocity data in the IMU data, combined with the last time pose data to predict the position, velocity and attitude, and the prediction result is used for re-projection error construction and nonlinear least squares optimization to improve the stability of the pose estimation.

8. The Unity-based WeChat Mini Program AR development framework of claim 1, wherein, The coordinate system alignment and AR space management module uses the RANSAC algorithm to detect a real plane, fits an initial plane by randomly selecting three non-collinear three-dimensional points, determines the optimal plane by counting the number of inliers, judges the plane type according to the plane normal vector, and generates anchor point data containing a unique anchor point ID, a position in the Unity world coordinate system and an attitude.

9. The Unity-based WeChat Mini Program AR development framework of claim 4, wherein, The virtual resource editing, loading and management module supports resource cache management: loaded virtual resource instances are cached locally, and are directly loaded next time the application is started; resources are automatically unloaded and memory is released when the resources are no longer used; and only resource package difference files are downloaded when the resources are updated.

10. Application of the Unity-based WeChat Mini Program AR development framework, characterized in that, The Unity-based WeChat mini-program AR development framework of any one of claims 1 to 9 is applied to realize the development and landing of cultural interaction, scene roaming and product display AR applications, and the application process includes a complete closed loop of user authentication, runtime initialization, data acquisition, pose estimation, coordinate alignment, resource loading, virtual-real fusion, interaction response and resource recycling.

Images