Heterogeneous framework fusion device and method for mixed reality or augmented reality

Abstract

This application provides a heterogeneous framework fusion device and method for mixed reality or augmented reality, belonging to the field of mixed reality and augmented reality technology, aiming to solve the problems of low data communication efficiency and high interaction latency caused by heterogeneous framework fusion. The solution includes a 3D content processing module, a native hardware interaction module, a cross-platform user interface module, and a heterogeneous framework communication adaptation module connecting the above modules. Each module sends data and instructions to the adaptation module, which performs format conversion and adaptation according to a unified data standard and communication protocol, and then distributes the processed data to the target module for scene rendering or interface updates. This application achieves efficient collaboration between heterogeneous frameworks by establishing a dedicated communication adaptation module as a hub, which can significantly reduce interaction latency and enhance system compatibility and stability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to the technical field of mixed reality and augmented reality. More particularly, the present application relates to a heterogeneous framework fusion device and method for mixed reality or augmented reality. BACKGROUND

[0002] With the popularity of mixed reality and augmented reality technology, it has become an industry hotspot to develop high-performance mixed reality or augmented reality applications on mobile terminals. Typical application development usually needs to integrate software frameworks of multiple different technology stacks: first, a three-dimensional content processing framework is needed to create, render and interact with complex virtual three-dimensional objects; second, a native hardware interaction framework is needed to efficiently access the underlying hardware of the device, such as cameras, inertial measurement units, etc., to obtain real-time environmental information and device pose; finally, in order to achieve rapid development and multi-platform deployment, developers often use a cross-platform user interface framework to build the two-dimensional interface and business logic of the application.

[0003] However, these three frameworks have natural differences in technical architecture, data format, communication protocol and running mechanism, and belong to heterogeneous frameworks. The fusion solutions in the prior art mostly stay at the level of simple upper interface calling, that is, the functions of another framework are called through an application programming interface in one framework. This way fails to fundamentally solve the communication barriers between frameworks, resulting in significant delay when data is transmitted between different frameworks, which needs to go through multiple format conversions and protocol parsing. In actual applications, this delay manifests as: when the user moves the device, the position of the virtual object updates lagging behind the changes in the real scene, resulting in drift and jitter; the user's operations on the interface need to wait for a long time before they are responded in the three-dimensional scene; and even due to data format incompatibility or synchronization errors, the application freezes or crashes. These problems seriously affect the core experience of mixed reality or augmented reality applications, i.e., the seamless integration of virtual and real, and constitute a difficult problem to be solved in current technical development. SUMMARY

[0004] The purpose of the present application is to provide a heterogeneous framework fusion device and method for mixed reality or augmented reality, aiming to solve the technical problems of low data communication efficiency, poor compatibility, high interaction delay and inconsistent rendering effect caused by fusing heterogeneous software frameworks in the prior art.

[0005] To achieve the above object, the application provides a heterogeneous framework fusion device for mixed reality or augmented reality, comprising: a three-dimensional content processing module, configured to process three-dimensional virtual content in mixed reality or augmented reality application, send scene data to a heterogeneous framework communication adaptation module, and receive hardware data provided by the heterogeneous framework communication adaptation module; a native hardware interaction module, configured to call a native application programming interface of a terminal device to access underlying hardware resources, collect original hardware data, and send the original hardware data to the heterogeneous framework communication adaptation module; a cross-platform user interface module, configured to build a two-dimensional user interface and front-end business logic of an application, send interface configuration data and cross-platform interaction instructions to the heterogeneous framework communication adaptation module, and receive scene rendering data and interaction results provided by the heterogeneous framework communication adaptation module; and the heterogeneous framework communication adaptation module, connected with the three-dimensional content processing module, the native hardware interaction module and the cross-platform user interface module, configured to receive data and instructions from the three-dimensional content processing module, the native hardware interaction module and the cross-platform user interface module, perform format conversion and protocol adaptation according to a preset unified data standard and communication protocol, and route and distribute processed data and instructions to corresponding target modules.

[0006] Optionally, the native hardware interaction module is configured to call a camera, a gyroscope and an accelerometer of the terminal device; and the original hardware data comprises real-time camera image data and device posture data.

[0007] Optionally, the scene data sent by the three-dimensional content processing module comprises model data of a three-dimensional model and interaction logic instructions; and the interface configuration data sent by the cross-platform user interface module comprises layout information of interface controls, and the cross-platform interaction instructions comprise operation instructions of a user on a two-dimensional user interface.

[0008] Optionally, the heterogeneous framework communication adaptation module adopts a JSON format as the unified data standard and adopts a WebSocket protocol as the communication protocol.

[0009] Optionally, the heterogeneous framework communication adaptation module adopts a Protocol Buffers data format as the unified data standard and adopts a gRPC framework as the communication protocol.

[0010] Optionally, the format conversion of the heterogeneous framework communication adaptation module comprises converting device posture data in the original hardware data into a three-dimensional coordinate and rotation format recognizable by the three-dimensional content processing module.

[0011] Optionally, the heterogeneous framework communication adaptation module is internally provided with an instruction queue with priority, which is used to process the received data and instructions according to the preset priority; wherein the processing priority of the original hardware data sent by the native hardware interaction module is higher than the processing priority of the interface configuration data and the cross-platform interaction instructions sent by the cross-platform user interface module.

[0012] Optionally, the heterogeneous framework communication adaptation module further comprises a data synchronization unit, which is used to attach a timestamp to the original hardware data with high priority, and generate compensation data through interpolation or extrapolation algorithm when detecting data discontinuity, so as to ensure the smoothness of the data stream distributed to the three-dimensional content processing module.

[0013] Optionally, the heterogeneous framework communication adaptation module adopts a mixed data transmission strategy, wherein: for control instructions and metadata, a unified data standard based on text is adopted for encapsulation and transmission; and for large binary data such as camera real-time image frames or model mesh data, direct binary stream is adopted for transmission.

[0014] To achieve the above-mentioned purpose, the application further provides a heterogeneous framework fusion method for mixed reality or augmented reality, comprising the following steps: collecting original hardware data by a native hardware interaction module, and sending the original hardware data to a heterogeneous framework communication adaptation module; generating scene data by a three-dimensional content processing module, and sending the scene data to the heterogeneous framework communication adaptation module; generating interface configuration data and cross-platform interaction instructions by a cross-platform user interface module, and sending them to the heterogeneous framework communication adaptation module; receiving the above-mentioned data and instructions by the heterogeneous framework communication adaptation module, performing unified format conversion and protocol adaptation, and distributing the processed data and instructions to the corresponding target module; rendering the scene by the three-dimensional content processing module according to the data received from the heterogeneous framework communication adaptation module; and / or updating the interface by the cross-platform user interface module according to the data received from the heterogeneous framework communication adaptation module.

[0015] Compared with the prior art, the application has the following beneficial effects:

[0016] 1. Significantly reduce the interaction delay and improve the real-time performance. The application sets up a dedicated heterogeneous framework communication adaptation module, which changes the originally scattered and inefficient many-to-many data conversion process into a centralized and efficient star-type communication mode. The module processes and routes the data, greatly reducing the conversion overhead and time cost of the data between different technology stacks, thereby effectively reducing the end-to-end delay from hardware data collection to scene rendering, and avoiding problems such as virtual object drift and lag.

[0017] 2. Enhanced System Compatibility and Stability. This application establishes a stable adaptation layer between different frameworks through a heterogeneous framework communication adaptation module, resolving compatibility issues caused by inconsistencies in data formats and protocols across frameworks. This decoupled design ensures that upgrading or replacing any module only requires adjusting the corresponding interface of the adaptation module, without affecting the overall system stability, thus enhancing the system's robustness and maintainability.

[0018] 3. Ensure consistent rendering across platforms. Because all data used for rendering undergoes standardized processing by the heterogeneous framework communication adaptation module, the 3D content processing module can receive uniformly formatted and accurately synchronized data input regardless of the terminal device it runs on. This ensures a high degree of consistency in the rendering effects of mixed reality or augmented reality scenes across different platforms, enhancing the user experience. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A schematic diagram of the system architecture of a heterogeneous framework fusion device provided in this application embodiment;

[0021] Figure 2 A flowchart illustrating a heterogeneous framework fusion method provided in this application embodiment;

[0022] Figure 3 This is a timing diagram showing the signaling interaction between the modules in the embodiments of this application;

[0023] Figure 4 This is a schematic diagram of the internal structure of a heterogeneous frame communication adapter module in one embodiment of this application;

[0024] Figure 5 This is a schematic diagram of a hybrid data transmission strategy in another embodiment of this application;

[0025] The reference numerals in the attached figures are explained as follows: 10-3D content processing module; 20-Native hardware interaction module; 30-Cross-platform user interface module; 40-Heterogeneous framework communication adaptation module; 41-Instruction receiving unit; 42-Priority queue; 43-Instruction processing thread; 44-Data synchronization unit; 45-JSON parser; 46-Binary data channel; 47-GPU processing unit. Detailed Implementation

[0026] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0027] Example 1

[0028] This embodiment provides a heterogeneous framework fusion device and method for mixed reality or augmented reality. Figure 1 This is a schematic diagram of the system architecture of a heterogeneous framework fusion device according to one embodiment of this application. The device can be deployed on terminal devices such as smartphones, tablets, and head-mounted displays. Figure 1 As shown, the device includes: a 3D content processing module 10, a native hardware interaction module 20, a cross-platform user interface module 30, and a heterogeneous framework communication adaptation module 40 as the core hub.

[0029] In one embodiment of this application, the device is deployed on an Apple iPhone or iPad running the iOS operating system. Specifically, the 3D content processing module 10 is developed based on the Unity3D engine (e.g., version 2021.3) and its development language is C#. The native hardware interaction module 20 is developed based on the iOS native development environment (e.g., using Xcode 13.0) and its development language is Objective-C. The cross-platform user interface module 30 is developed based on the Ionic framework (e.g., version 6.0) and its development language is TypeScript, relying on the underlying Web technology stack (HTML, CSS, JavaScript). Correspondingly, the heterogeneous framework communication adaptation module 40 is a custom-developed communication middleware that spans the above three technology stacks. In this embodiment, it uses JSON as the unified data exchange standard and WebSocket as the underlying communication protocol.

[0030] The 3D content processing module 10 is primarily responsible for processing all 3D virtual content in mixed reality or augmented reality applications. Specifically, this module utilizes the rendering capabilities of the Unity3D engine to load, render, and manage 3D models. For example, in an augmented reality home furnishing shopping guide application, this module can render a model of a virtual sofa. This module also handles interaction logic related to the 3D scene, such as allowing users to zoom or rotate the virtual sofa using gestures. The module generates scene data (e.g., models, model positions, lighting information, etc. included in the initial scene) and encapsulates this scene data before sending it to the heterogeneous frame communication adapter module 40. To achieve precise alignment between virtual objects and the real world, the 3D content processing module 10 needs to receive, in real-time, formatted hardware data provided by the heterogeneous frame communication adapter module 40, especially the device's posture data, and adjust the position and rotation of its internal virtual camera accordingly, so that the user perceives the virtual object as being "fixed" in the real environment.

[0031] The native hardware interaction module 20 acts as a bridge between the system and the underlying hardware of the terminal device. On the iOS platform, this module interacts with the hardware by calling native application programming interfaces (APIs) such as the AVFoundation framework for accessing the camera and the CoreMotion framework for accessing the inertial measurement unit. This module is responsible for collecting the raw hardware data necessary to implement mixed reality or augmented reality functions. As an optional implementation, this module can continuously collect real-time image data from the camera to be used as the background for the augmented reality scene. Simultaneously, this module collects device attitude data calculated by fusion of gyroscope and accelerometer data at a high frequency (e.g., 60 times per second), which is typically represented as the device's orientation in space in the form of quaternions or rotation matrices. All collected raw hardware data, including real-time camera image data and device attitude data, is sent to the heterogeneous framework communication adapter module 40 for further processing and distribution.

[0032] The cross-platform user interface module 30 is primarily responsible for building the application's two-dimensional user interface (UI) and handling front-end business logic. Using a cross-platform framework like Ionic enables "develop once, deploy across multiple platforms," ​​thereby improving development efficiency. In this embodiment, this module is responsible for building the non-3D parts of the application, including but not limited to the product list, settings page, and user login interface. This module sends interface configuration data (e.g., layout information of controls on the interface) and user cross-platform interaction commands (e.g., a user clicking on a product in the product list) to the heterogeneous framework communication adaptation module 40. Conversely, this module also receives processed data from the heterogeneous framework communication adaptation module 40, such as interaction results returned from the 3D scene (e.g., successful model loading) or brief scene rendering data (e.g., the name of the currently displayed model), and updates the user interface accordingly.

[0033] The heterogeneous framework communication adaptation module 40 is the core of this application embodiment. As an independent logical layer, this module connects and communicates with the three modules with vastly different technical architectures, solving the problem of low efficiency or inability to communicate directly due to differences in data formats, communication protocols, and operating mechanisms. In this embodiment, the heterogeneous framework communication adaptation module 40 is specifically implemented as follows: a WebSocket client is implemented in each of the 3D content processing module 10 (C# environment), the native hardware interaction module 20 (Objective-C environment), and the cross-platform user interface module 30 (TypeScript environment). Simultaneously, an independent WebSocket server process can be established, or one of the modules (e.g., the native hardware interaction module 20) can also serve as the server. All modules connect to this central service via WebSocket.

[0034] This module operates based on a pre-defined unified data standard and communication protocol. In this embodiment, the unified data standard is defined as JSON format. All data transmitted between modules, whether it is device posture, UI click events, or model information, must first be encapsulated into a JSON object with a specific structure. For example, a posture data JSON object sent from the native hardware interaction module 20 to the heterogeneous framework communication adapter module 40 can be represented as:

[0035] {

[0036] "type": "pose_update",

[0037] "timestamp": 1678886400123,

[0038] "payload": {

[0039] "position": {"x": 0.1, "y": 1.5, "z": -2.0},

[0040] "rotation": {"x": 0.0, "y": 0.707, "z": 0.0, "w": 0.707}

[0041] }

[0042] }

[0043] An interactive command JSON object issued from the cross-platform user interface module 30 can be represented as:

[0044] {

[0045] "type": "ui_command",

[0046] "command": "show_model",

[0047] "payload": {

[0048] "model_id": "sofa_001

[0049] }

[0050] }

[0051] When the heterogeneous framework communication adaptation module 40 receives such a JSON message, it performs format conversion and protocol adaptation. For example, when it receives attitude data from the native hardware interaction module 20, the heterogeneous framework communication adaptation module 40 parses the JSON object. Understandably, since the 3D content processing module 10 (Unity3D) typically uses its custom Vector3 and Quaternion data structures to represent position and rotation, directly transmitting a JSON string is inefficient. Therefore, the core conversion logic of the heterogeneous framework communication adaptation module 40 is to convert the x, y, z, w values ​​representing position and rotation in the JSON into data types that can be directly used by the C# code of the 3D content processing module 10. This conversion can occur on the server side of the heterogeneous framework communication adaptation module 40, or immediately after the WebSocket client on the Unity side receives the JSON string. Converting device attitude data into a 3D coordinate and rotation format recognizable by the 3D content processing module is a crucial step in ensuring accurate alignment between the virtual and real worlds. After processing, the heterogeneous framework communication adaptation module 40 routes and distributes the converted data or instructions to the corresponding target modules. For example, pose data may be forwarded to the 3D content processing module 10, and UI commands may also be forwarded to the 3D content processing module 10 to trigger scene changes.

[0052] The following will combine Figure 2 and Figure 3 Taking the specific working process of an augmented reality (AR) shopping guide application as an example, the method steps of this application embodiment will be described in detail. Among them, Figure 2 A flowchart illustrating a heterogeneous framework fusion method provided in this application embodiment. Figure 3 This is a timing diagram showing the signaling interaction between the modules in the embodiments of this application.

[0053] This method begins with three parallel preparation phases. First, in step S101, raw hardware data is acquired. The native hardware interaction module 20 continuously acquires real-time device posture data (including position and rotation information) by calling the iOS CoreMotion framework, and encapsulates this data into the aforementioned defined JSON format. This data is then sent frequently to the heterogeneous framework communication adaptation module 40 via a WebSocket connection, forming a sequence as follows: Figure 3 The continuous data stream shown.

[0054] Secondly, in step S102, MR / AR scene data is constructed. MR stands for Mixed Reality. The 3D content processing module 10 loads a basic AR scene upon application startup, which may contain pre-defined virtual objects or environmental information. This initial scene data, such as model data of a virtual ground (vertices, textures, etc.), is also encapsulated in JSON format and sent to the heterogeneous framework communication adaptation module 40, which can be used to notify other modules that the scene is ready.

[0055] Third, in step S103, interface configuration data is generated. The cross-platform user interface module 30 constructs and displays the application's two-dimensional user interface, such as a product list showing all available furniture for preview. The layout information and product metadata (such as names and thumbnails) of this list are themselves part of the interface configuration data, and some of this metadata may also need to be synchronized with other modules through the heterogeneous framework communication adaptation module 40.

[0056] Subsequently, format conversion and protocol adaptation are performed in step S104. The heterogeneous framework communication adaptation module 40, acting as the data hub, continuously receives data and instructions from the three modules mentioned above. For example, if a user clicks the "Preview Sofa" button on the product list in the cross-platform user interface module 30, the cross-platform user interface module 30 will generate a JSON instruction of type ui_command as shown above and send it to the heterogeneous framework communication adaptation module 40 via WebSocket. Figure 3 As shown, this is an event-driven signaling.

[0057] Upon receiving the instruction, the heterogeneous frame communication adaptation module 40 parses and routes it, recognizing that it is an instruction requesting the 3D content processing module 10 to display the model. It then forwards the instruction in JSON format to the 3D content processing module 10. Simultaneously, the heterogeneous frame communication adaptation module 40 continuously provides the latest pose data, converted from the native hardware interaction module 20, to the 3D content processing module 10.

[0058] The process then proceeds to steps S105 and S106. In this example, step S105 performs interface adaptation and rendering preparation. This may involve the cross-platform user interface module 30 displaying a loading prompt on the interface after receiving the "model loading" status forwarded by the heterogeneous framework communication adaptation module 40.

[0059] Finally, scene rendering and data application are performed in step S106. After receiving the `show_model` command from the heterogeneous framework communication adaptation module 40, the C# script of the 3D content processing module 10 loads the corresponding 3D sofa model resource based on `model_id`. Then, in each frame update loop (Update function) of Unity, this module obtains the latest device pose data (converted to Vector3 and Quaternion) from the heterogeneous framework communication adaptation module 40 and uses this data to update the position and pose of the main camera in its scene. Thus, when the user moves their phone, the real-world background in the camera's field of view changes, and the synchronous change of the virtual camera makes the rendered sofa model appear to be stably placed in a real-world location. The final rendering result, i.e., the camera view with the virtual sofa, is presented in the view hierarchy managed by the cross-platform user interface module 30 through specific techniques (such as using Unity's rendered view as a subview of the native view) for the user to view.

[0060] Through the aforementioned apparatus and method, this embodiment achieves the integration of three heterogeneous technology frameworks, enabling a single application to quickly build a UI using cross-platform frameworks, obtain high-performance hardware data using native APIs, and render complex 3D scenes using a professional engine. The core heterogeneous framework communication adaptation module 40, through unified data standards and communication protocols, reduces cross-framework communication latency and improves system stability and compatibility.

[0061] Example 2

[0062] This embodiment aims to illustrate that the technical solution proposed in this application has broad applicability and is not limited to a specific combination of technology stacks. As an optional implementation, this embodiment deploys the device on a terminal device running the Android operating system and adopts a mainstream technology framework different from that of Embodiment 1.

[0063] In this embodiment, the 3D content processing module 10 uses Unreal Engine, primarily developed in C++ or Blueprint Visualization Script. The native hardware interaction module 20 is developed natively for Android, using Java or Kotlin, and calls the ARCore library provided by Google to obtain environmental awareness and device tracking capabilities. The cross-platform user interface module 30 uses the Flutter framework and the Dart language to build a cross-platform user interface with high performance and a native experience.

[0064] A key difference between this embodiment and Embodiment 1 is that, in pursuit of higher performance, the heterogeneous framework communication adaptation module 40 in this embodiment uses gRPC as the communication framework and Protocol Buffers as the unified data standard. gRPC is a high-performance, open-source, general-purpose remote procedure call framework based on the HTTP / 2 protocol, supporting advanced features such as bidirectional streaming and header compression. Protocol Buffers, on the other hand, is a language- and platform-independent, scalable method for serializing structured data, and its serialized data is typically smaller and faster than JSON.

[0065] The specific implementation process is as follows: First, a .proto file is needed to define the service interface and data structure. For example, an ARData service can be defined, along with message types such as PoseData and RenderCommand.

[0066] syntax = "proto3";

[0067] service ARData {

[0068] / / Server-side streaming RPC, native module continuously pushes posture data

[0069] rpc StreamPoseData (stream PoseData) returns (stream Ack);

[0070] / / Simple client-side RPC, UI module sends rendering commands

[0071] rpc SendRenderCommand (RenderCommand) returns (Ack);

[0072] }

[0073] message PoseData {

[0074] int64 timestamp = 1;

[0075] message Vector3 {

[0076] float x = 1;

[0077] float y = 2;

[0078] float z = 3;

[0079] }

[0080] message Quaternion {

[0081] float x = 1;

[0082] float y = 2;

[0083] float z = 3;

[0084] float w = 4;

[0085] }

[0086] Vector3 position = 2;

[0087] Quaternion rotation = 3;

[0088] }

[0089] message RenderCommand {

[0090] string command_name = 1;

[0091] string model_id = 2;

[0092] }

[0093] message Ack {

[0094] bool success = 1;

[0095] string message = 2;

[0096] }

[0097] After defining the .proto file, the Protocol Buffers compiler can generate corresponding client and server stub code for C++ (for Unreal Engine), Java (for native Android), and Dart (for Flutter).

[0098] The core logic of the heterogeneous framework communication adaptation module 40 is implemented on the gRPC server, which can be used as an independent Android background service process.

[0099] The work process is as follows:

[0100] 1. In the native hardware interaction module 20 (Android native code), the device's six degrees of freedom tracking data, i.e., precise position and orientation, is obtained through the ARCore API. After obtaining the data, a PoseData ProtocolBuffers message object is populated, and this strongly typed binary message is pushed to the gRPC server, which serves as the heterogeneous framework communication adaptation module 40, via a gRPC client streaming call. Due to the use of binary serialization, this process eliminates the need for time-consuming string concatenation and parsing, resulting in high efficiency.

[0101] 2. In the cross-platform user interface module 30 (Flutter application), when the user performs an operation (such as clicking a button), the Flutter code creates a RenderCommand Protocol Buffers message object, fills in the command name and parameters, and then sends the message to the heterogeneous framework communication adaptation module 40 through a simple RPC call of gRPC.

[0102] 3. After receiving the PoseData message, the heterogeneous framework communication adaptation module 40 (gRPC server) can forward it to the Unreal Engine module via another gRPC channel with almost no conversion because its data structure is already strongly typed. Similarly, it forwards the RenderCommand message directly upon receiving it.

[0103] 4. In the 3D content processing module 10 (Unreal Engine), it acts as a gRPC client, continuously receiving PoseData messages from the heterogeneous framework communication adaptation module 40 via a server-side streaming RPC. Whenever new pose data is received, the C++ code directly uses it to update the transformations of the camera components in the scene. Simultaneously, this module also monitors and receives RenderCommand messages, executing corresponding actions based on the instructions, such as loading and displaying specific 3D models.

[0104] Compared to Example 1, this example employs Protocol Buffers and gRPC, resulting in smaller data transmission payloads and faster serialization and deserialization speeds. Furthermore, gRPC, based on HTTP / 2, offers lower latency and higher efficiency when processing high-frequency streaming data (such as pose data). This solution further reduces end-to-end latency from hardware acquisition to scene rendering, making the integration of virtual objects with reality more stable and seamless, particularly suitable for complex mixed reality or augmented reality applications with extremely high real-time requirements. This example demonstrates that the core idea of ​​this application—decoupling and connecting heterogeneous frameworks through a central communication adapter module—has good universality.

[0105] Example 3

[0106] This embodiment, based on the technology stack of Embodiment 1 (Unity3D, native iOS, Ionic, JSON / WebSocket), optimizes the internal processing mechanism of the heterogeneous framework communication adaptation module 40, aiming to solve the possible conflicts and timing problems between different data streams, thereby further improving the stability of the system and the user experience.

[0107] Please see Figure 4 The figure shows the internal structure of the heterogeneous frame communication adapter module 40 in this embodiment. As shown in the figure, the optimized heterogeneous frame communication adapter module 40 can be logically divided into several core components: instruction receiving unit 41, priority queue 42, instruction processing thread 43, and data synchronization unit 44.

[0108] The instruction receiving unit 41 serves as the entry point for the heterogeneous framework communication adaptation module 40. It monitors and receives all data and instructions from the 3D content processing module 10, the native hardware interaction module 20, and the cross-platform user interface module 30 via a WebSocket connection. Upon receiving a JSON message, this unit first parses the `type` field in the message header to identify its data type and origin.

[0109] After identification, the data or instructions are not processed immediately, but are instead sent to a priority queue 42. This queue is one of the core components of the optimization scheme in this embodiment. The system assigns different priorities to different types of tasks according to preset rules. A typical priority strategy is as follows:

[0110] - High priority: Device pose data (pose_update) sent by the native hardware interaction module 20. This data is crucial for ensuring the real-time performance of AR tracking and must be processed and forwarded with the highest priority.

[0111] - Medium priority: Interactive events that require a UI response, sent by the 3D content processing module 10 (e.g., a user clicking on a virtual object in a 3D scene), or key business instructions (e.g., a purchase instruction) sent by the cross-platform user interface module 30.

[0112] - Low priority: Non-real-time UI update requests (such as refreshing the news list) sent by the cross-platform user interface module 30, or background data such as logs and statistics sent by the 3D content processing module 10.

[0113] Instruction processing thread 43 consists of one or more background worker threads that continuously retrieve tasks from the head of priority queue 42 for execution. Due to the characteristics of the queue, high-priority tasks are always retrieved first. In other words, even under high system load and with a large number of low-priority UI update requests queuing, critical pose data processing tasks will not be blocked, thus ensuring the smooth tracking of virtual objects in the AR scene.

[0114] The data synchronization unit 44 is another optimization for high-priority data (especially attitude data). High-frequency attitude data streams are sensitive to network stability; network jitter or processing delays can lead to data frame loss or out-of-order delivery, which manifests as momentary jitter or "jumping" of virtual objects in the user experience. The data synchronization unit 44 aims to mitigate this problem, and its operating mechanism is as follows: 1. When the instruction receiving unit 41 receives a high-priority attitude data packet, it first passes it to the data synchronization unit 44. 2. The data synchronization unit 44 appends a high-precision local timestamp to each frame of valid attitude data. 3. Before placing the attitude data into the priority queue 42, the data synchronization unit 44 checks the continuity of the data stream. For example, by comparing the timestamp interval between the current frame and the previous frame, if the interval is much larger than the expected frame interval (e.g., expected 16.6 milliseconds, actually exceeding 30 milliseconds), it can be determined that frame loss has occurred. 4. When data discontinuity is detected, the data synchronization unit 44 initiates a compensation algorithm. One feasible compensation method is linear extrapolation. For example, based on the attitude changes (differences in displacement and rotation) of the first two frames, one or more intermediate compensated attitude data frames are predicted and generated, and inserted into the data stream. The extrapolation formula can be expressed as: P_t = P_(t-1) + (P_(t-1) - P_(t-2)) where P represents the attitude (including position and rotation), and t represents the frame. Extrapolation of rotation can be implemented using an extension of spherical linear interpolation (Slerp). 5. The smoothed attitude data stream is then placed into priority queue 42.

[0115] By introducing a priority queue 42 and a data synchronization unit 44, the heterogeneous framework communication adaptation module 40 in this embodiment possesses the ability to proactively perform traffic scheduling and data quality assurance, rather than simply passively performing data conversion and forwarding. This optimization improves the stability of the AR experience; even when users quickly move their devices or in poor network conditions, virtual objects can maintain smooth and stable tracking effects, resulting in a better user experience compared to a simple command forwarding mechanism.

[0116] Example 4

[0117] Based on the technology stack of Embodiment 1, this embodiment optimizes the data format and transmission method of the heterogeneous framework communication adaptation module 40 according to the transmission characteristics of different types of data, and adopts a hybrid data transmission strategy.

[0118] In mixed reality or augmented reality applications, the types of data transmitted between modules vary significantly. One type consists of small, highly structured control commands and metadata, such as "display model" and "user ID." The other type comprises massive blocks of binary data, typically such as real-time camera image frames (a single 1080p YUV image can be several megabytes) obtained from native modules and large model mesh data loaded from 3D modules. If all data is encapsulated using JSON, encoding it into Base64 strings when processing large blocks of binary data will increase the data volume by approximately 33%, and decoding and parsing at the receiving end will consume significant CPU resources, causing latency.

[0119] To address this issue, this embodiment optimizes the data processing logic of the heterogeneous framework communication adaptation module 40. Please refer to [link / reference]. Figure 5 This figure is a schematic diagram of the hybrid data transmission strategy in this embodiment. When receiving and sending data, the heterogeneous framework communication adaptation module 40 selects different processing paths according to the data type.

[0120] The hybrid data transmission strategy is as follows:

[0121] For control commands and metadata, the unified text-based data standard, JSON, continues to be used for encapsulation and transmission via WebSocket text mode. When the heterogeneous framework communication adapter module 40 receives a text frame, it sends it to the JSON parser 45 for processing. JSON format offers good readability, is easy to debug and extend, and is suitable for handling this type of structured small data.

[0122] For large blocks of binary data such as real-time camera image frames or model mesh data, a direct binary stream is used for transmission via WebSocket's binary mode. That is, after the native hardware interaction module 20 acquires a YUV format camera image frame, it no longer encodes it as a Base64 string and places it in JSON; instead, it directly sends the raw byte array as the binary message body of the WebSocket. When the heterogeneous framework communication adaptation module 40 receives the binary frame, the data enters a dedicated binary data channel 46.

[0123] Transmission via binary data channel 46 avoids the performance overhead and data bloat associated with encoding / decoding. As a further optimization, this embodiment also introduces support for GPU-accelerated conversion. A GPU processing unit 47 can be integrated into the heterogeneous framework communication adapter module 40. This is a logical unit responsible for calling the GPU computing interfaces provided by the platform, such as Metal Performance Shaders on iOS or OpenGL Shaders (or Vulkan) on Android.

[0124] The process is as follows: When the native hardware interaction module 20 acquires a frame of YUV format camera image and sends it to the heterogeneous frame communication adapter module 40 via binary data channel 46, the heterogeneous frame communication adapter module 40 can directly (or through zero-copy / quasi-zero-copy technology) submit this memory data to the GPU processing unit 47. The GPU processing unit 47 executes a pre-written computational shader specifically responsible for efficiently converting the image in the YUV color space into RGB or RGBA format textures required by the 3D content processing module 10 (Unity3D). This conversion process is completed entirely on the GPU, which is much faster than the CPU and avoids data copying back and forth between CPU and GPU memory. After the conversion is complete, the generated texture can be directly used by the 3D content processing module 10 for rendering the scene background.

[0125] By adopting this hybrid data transmission strategy and combining it with GPU acceleration, this embodiment optimizes the processing path for large amounts of data (especially camera video streams) in the system, avoids the performance bottleneck of JSON processing of large data, reduces the latency of data transmission and format conversion, and allows CPU resources to be used for more core logic calculations, thereby making the background screen of AR applications smoother and improving the overall performance and response speed.

[0126] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A heterogeneous framework fusion device for mixed reality or augmented reality, characterized by, The application comprises: a three-dimensional content processing module for processing three-dimensional virtual content in a mixed reality or augmented reality application, sending scene data to a heterogeneous framework communication adaptation module, and receiving hardware data provided by the heterogeneous framework communication adaptation module; a native hardware interaction module for calling a native application programming interface of a terminal device to access underlying hardware resources, collecting raw hardware data, and sending the raw hardware data to the heterogeneous framework communication adaptation module; a cross-platform user interface module for building a two-dimensional user interface and front-end business logic of an application, sending interface configuration data and cross-platform interaction instructions to the heterogeneous framework communication adaptation module, and receiving scene rendering data and interaction results provided by the heterogeneous framework communication adaptation module; and the heterogeneous framework communication adaptation module connected to the three-dimensional content processing module, the native hardware interaction module, and the cross-platform user interface module, for receiving data and instructions from the three-dimensional content processing module, the native hardware interaction module, and the cross-platform user interface module, performing format conversion and protocol adaptation according to a preset unified data standard and communication protocol, and routing and distributing processed data and instructions to corresponding target modules.

2. The apparatus of claim 1, wherein, The native hardware interaction module is configured to call a camera, a gyroscope, and an accelerometer of a terminal device, and the raw hardware data includes real-time camera image data and device posture data.

3. The apparatus of claim 1, wherein, The scene data sent by the three-dimensional content processing module includes model data of a three-dimensional model and interaction logic instructions, and the interface configuration data sent by the cross-platform user interface module includes layout information of interface controls, and the cross-platform interaction instructions include operation instructions of a user on a two-dimensional user interface.

4. The apparatus of claim 1, wherein, The heterogeneous framework communication adaptation module adopts JSON as the unified data standard and WebSocket as the communication protocol.

5. The apparatus of claim 1, wherein, The heterogeneous framework communication adaptation module adopts ProtocolBuffers as the unified data standard and gRPC as the communication protocol.

6. The apparatus of claim 1, wherein, The format conversion of the heterogeneous framework communication adaptation module includes converting device posture data in the raw hardware data into a three-dimensional coordinate and rotation format recognizable by the three-dimensional content processing module.

7. The apparatus of claim 1, wherein, The heterogeneous framework communication adaptation module is internally provided with an instruction queue with priorities, for processing received data and instructions according to preset priorities, wherein a processing priority of raw hardware data sent by the native hardware interaction module is higher than processing priorities of interface configuration data and cross-platform interaction instructions sent by the cross-platform user interface module.

8. The apparatus of claim 7, wherein, The heterogeneous framework communication adaptation module further comprises a data synchronization unit for attaching a timestamp to high-priority raw hardware data, and generating compensation data through an interpolation or extrapolation algorithm when detecting data discontinuity, to ensure smoothness of a data stream distributed to the three-dimensional content processing module.

9. The apparatus of claim 1, wherein, The heterogeneous framework communication adaptation module adopts a hybrid data transmission strategy, wherein: for control instructions and metadata, a text-based unified data standard is adopted for encapsulation and transmission; Direct binary stream is adopted for transmission of real-time camera image frames or model mesh data.

10. A heterogeneous framework fusion method for mixed reality or augmented reality, characterized by, The method comprises the following steps: The native hardware interaction module collects original hardware data and sends the original hardware data to the heterogeneous framework communication adaptation module; The three-dimensional content processing module generates scene data and sends the scene data to the heterogeneous framework communication adaptation module; The cross-platform user interface module generates interface configuration data and cross-platform interaction instructions and sends them to the heterogeneous framework communication adaptation module; The heterogeneous framework communication adaptation module receives the above data and instructions, performs unified format conversion and protocol adaptation, and distributes the processed data and instructions to the corresponding target modules; The three-dimensional content processing module performs scene rendering according to the data received from the heterogeneous framework communication adaptation module; and / or The cross-platform user interface module performs interface updating according to the data received from the heterogeneous framework communication adaptation module.

Images