XR application editing methods, devices, and storage media

Abstract

Embodiments of this application provide a method for editing XR applications, a device, and a storage medium. In embodiments of this application, a runtime container is integrated into a content generator, and application content for an XR application is generated based on the runtime environment provided by the runtime container. When application content is generated, an XR application is generated based on the application content and the runtime container, and the runtime container is executed between the application content and the operating system, providing the runtime environment to the application content. By adding a runtime container between the application content and the operating system, the strong dependency between the execution of the application content and the operating system is eliminated, enabling the development and execution of cross-platform and cross-device XR applications. This simplifies and facilitates the development of XR applications, solving the problems of XR application development and cross-device execution.

Description

Technical Field

[0001] (Cross - reference to related applications) This application claims the benefit of Chinese Patent Application No. 2023106207762, filed on May 26, 2023, with the title "Editing Method, Device, and Storage Medium for XR Applications", and all of its contents are incorporated herein by reference in their entirety.

[0002] This application relates to the technical field of the Internet, and in particular, to an editing method, device, and storage medium for XR applications.

Background Art

[0003] Extended Reality (XR) is a general term for Augmented Reality (AR) and Virtual Reality (VR). VR is a technology that directly provides participants with sensations such as vision, hearing, and touch generated by a computer, enabling interactive observation and operation of the virtual world by the participants. AR is a technology that skillfully combines virtual information and the real world. By using various technical means such as multimedia, 3D modeling, intelligent interaction, and sensing, it simulates the real world with virtual information such as characters, images, 3D models, voices, and videos generated by a computer, and applies it to the real world to achieve an "expansion" of the real world.

[0004] With the popularization of XR technology, the supply of XR applications based on XR technology has become an issue that requires urgent solution. In the prior art, there are game development engines for three - dimensional (3D) game developers and two - dimensional (2D) application development engines for terminal research and development engineers, but these development engines are not for the development of XR applications. Therefore, there is an urgent need for development means that can develop XR applications.

Summary of the Invention

[0005] Multiple aspects of this application provide methods, devices, and storage media for editing XR applications to solve the development problems of XR applications.

[0006] The embodiment of this application provides a runtime container necessary for generating the XR application, for providing a runtime environment during the generation and execution processes of the XR application, Internally, the runtime container is integrated, and a content generator is provided for generating application content for the XR application based on the runtime environment provided by the runtime container. Includes an application generation platform for generating the XR application based on the application content and the runtime container, The present invention provides an XR application development system for terminal development, wherein, during the execution process of the XR application, the runtime container is executed between the application content and the operating system of the target device on which the XR application is located, and the application content is executed in the runtime environment provided by the runtime container.

[0007] The embodiment of this application provides a method for developing an XR application for terminal development, comprising the steps of: developing in advance a runtime container necessary for generating an XR application to provide a runtime environment during the generation and execution processes of the XR application; generating application content for the XR application based on the runtime environment provided by the runtime container in a content generator in which the runtime container is integrated; and generating the XR application based on the application content and the runtime container, wherein during the execution process of the XR application, the runtime container is executed between the application content and the operating system of the target device on which the XR application is located, and the application content is executed in the runtime environment provided by the runtime container.

[0008] An embodiment of this application is a content generator that provides a runtime environment during the generation or execution process of an XR application, and which has a runtime container necessary for generating the XR application internally integrated. A 3D scene effect editing module for obtaining 3D scene resource data by loading static resources of a 3D scene in response to the generation operation of an XR application, and editing dynamic information of the 3D scene for the static resources of the 3D scene based on the runtime environment provided by the runtime container, The system further comprises a 3D application logic editing module for editing application logic code for resource data of the 3D scene and obtaining application content of the XR application, based on the runtime environment provided by the runtime container, The XR application is formed by the application content and the runtime container, and the runtime container in the XR application provides a runtime environment for the application content during the execution process of the XR application, further providing a content generator.

[0009] Embodiments of the present application provide a method for editing an XR application applied to a content generator that integrates a runtime container for providing a runtime environment during the generation or execution process of an XR application, comprising the steps of: loading static resources of a 3D scene in response to an XR application generation operation; obtaining resource data of a 3D scene by adding dynamic information of the 3D scene to the static resources of the 3D scene based on a runtime environment provided by the runtime container; and obtaining application content of the XR application by editing application logic code for the resource data of the 3D scene based on a runtime environment provided by the runtime container, wherein the XR application is formed by the application content and the runtime container, and the runtime container in the XR application provides a runtime environment to the application content during the execution process of the XR application.

[0010] An embodiment of this application is a runtime container for providing a runtime environment during the generation and execution processes of an XR application, comprising a framework layer, a runtime environment layer, and a software library layer. The software library layer includes multiple types of software engines and provides a second type of application programming interface API for the multiple types of software engines, thereby obtaining a first type of API through encapsulation based on the second type of API. The framework layer provides the application developer with the first type of API so that the application developer can create the application logic code for the XR application based on the first type of API, and during the generation or execution process of the XR application, it senses a target event, executes at least one function entity of the first type of API corresponding to the target event, and provides the runtime environment layer with a target API, which is a second type of API encapsulated in the function entity. The runtime environment layer further provides a runtime container that, depending on the target API, calls the corresponding software engine in the software library layer to respond to the target event.

[0011] Embodiments of the present application further provide a method for executing an XR application, which includes the steps of: providing a runtime environment for application content in an XR application by operating a runtime container in the XR application in response to a trigger operation for executing the XR application; the runtime container performing a step between the application content and the operating system of a target device on which the XR application is located; and executing the XR application by executing the application content based on the runtime environment provided by the runtime container.

[0012] Embodiments of this application provide a method for executing XR content applied to a host application incorporating an XR runtime container, further comprising the steps of: displaying access portal information for the XR content during the execution process of the host application; operating the XR runtime container and acquiring the XR content in response to a trigger operation on the access portal information; and executing the XR content based on a runtime environment provided by the XR runtime container.

[0013] Embodiments of this application provide an electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor is coupled with the memory and executes the computer program, thereby realizing steps in a method for editing an XR application, a method for executing an XR application, a method for developing an XR application, or a method for executing XR content provided by embodiments of this application.

[0014] Embodiments of this application provide a computer-readable storage medium in which a computer program is stored, wherein when the computer program is executed by a processor, the processor enables the processor to implement steps in a method for editing an XR application, a method for executing an XR application, a method for developing an XR application, or a method for executing XR content provided by embodiments of this application.

[0015] In the embodiment of this application, a runtime container is integrated into the content generator, and application content for an XR application is generated based on the runtime environment provided by the runtime container. When application content is generated, an XR application containing the application content and the runtime container is generated based on the application content and the runtime container. In the execution process of an XR application developed based on the runtime container, the runtime container is executed between the application content and the operating system, and the runtime container provides the runtime environment to the application content, so that the application content is executable in the runtime environment. By adding a runtime container between the application content and the operating system, the strong dependency between the execution of the application content and the operating system is eliminated, enabling the development and execution of cross-platform and cross-device XR applications. This simplifies and facilitates the development of XR applications, solving the problems of XR application development and cross-device execution. [Brief explanation of the drawing]

[0016] The drawings described herein are provided to provide a further understanding of this application and constitute part of this application; however, the schematic embodiments and descriptions herein are for interpretive purposes only and do not constitute an inappropriate limitation to this application. [Figure 1] This is a schematic diagram of the structure of an XR application development system for terminal development provided by the embodiment of this application. [Figure 2] This is one implementation configuration of the runtime container provided by this embodiment. [Figure 3] This is a schematic diagram of the relationships between the three layers of the runtime container provided by the embodiment of this application. [Figure 4]It is a schematic diagram of the runtime framework of the XR application provided by the embodiments of the present application. [Figure 5] It is a schematic diagram of a 3D scene in which a 3D scene is incorporated into the host 2D application provided by the embodiments of the present application. [Figure 6a] It is a schematic flowchart of the initialization of the XR application provided by the embodiments of the present application. [Figure 6b] It is a schematic flowchart of the user triggering a function call provided by the embodiments of the present application. [Figure 6c] It is a schematic flowchart of the user triggering an event callback provided by the embodiments of the present application. [Figure 6d] It is a schematic structural diagram of the content generator provided by the embodiments of the present application. [Figure 7a] It is a schematic flowchart of the development method of the XR application provided by the embodiments of the present application. [Figure 7b] It is a schematic flowchart of the editing method of the XR application provided by the embodiments of the present application. [Figure 8] It is a schematic flowchart of the execution method of the XR application provided by the embodiments of the present application. [Figure 9] It is a schematic structural diagram of the electronic device provided by the embodiments of the present application.

Embodiments for Carrying Out the Invention

[0017] To make the purpose, technical means, and advantages of the present application clearer, the technical means of the present application will be clearly and completely described below by referring to the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments that can be obtained by those skilled in the art without creative labor shall fall within the protection scope of the present application.

[0018] Furthermore, the information described in this application (including, but not limited to, user device information and user personal information), data (including, but not limited to, data for analysis, data for storage, data for display), and signals are all information and data authorized by the user or fully authorized by each relevant party, and the collection, use, and processing of the relevant data must comply with the relevant laws, regulations, and standards of the relevant region, and a corresponding operational portal must be provided for the user to select, authorize, or reject such data.

[0019] With the spread of XR technology, developing XR applications has become a challenge for conventional technologies. In particular, before the widespread adoption of XR devices, XR applications mainly consisted of 3D scenes integrated into 2D applications. Consequently, for a long time, XR application development was primarily carried out by terminal research and development engineers. Terminal research and development engineers are engineers who develop 2D applications for mobile devices such as cell phones, and include, but are not limited to, client engineers and front-end engineers.

[0020] To address the challenges of developing XR applications, application developers, including terminal research and development engineers, have attempted to develop XR applications using game engines. A game engine is a framework for rapidly developing games, and commonly used game engines include, but are not limited to, Unity and UE. Game engines are highly functional tools designed for game development, including various functional modules such as rendering, physics, and animation, and it is possible to develop complete and complex XR applications based on a game engine. However, XR application development based on game engines has the following shortcomings.

[0021] First, for terminal research and development engineers, there are the following difficulties in shifting their focus to game research and development.

[0022] (1) Game engines place high demands on terminal research and development engineers, requiring them to have a high level of understanding of 3D graphics, making them difficult to work with.

[0023] (2) It is necessary to be proficient in game engines, and game engines such as UE / Unity contain hundreds of header files and thousands of Application Programming Interfaces (APIs), making it difficult to become proficient in them.

[0024] (3) Game engine integrated development environments (IDEs), such as UE Blueprint and Unity Editor toolchains, are designed independently and differ significantly from application development IDEs, making it difficult for terminal research and development engineers to adapt to them in a short period of time.

[0025] Next, since the game engine is not designed for embedded apps, the following problems may arise when developing applications that incorporate 3D scenes into 2D applications.

[0026] (1) Multiple Instances: The defining characteristic of 2D applications is the switching between different pages. Pages are isolated from each other, and each page is an independent instance to support state cache recovery and fallback. However, the game engine operates in monopoly mode, lacking lifecycle management. This makes it difficult to reclaim resources after page switching, leading to resource waste. Memory resources are more likely to reach the upper limit of the application's available resources, resulting in application crashes.

[0027] (2) User Interface (UI) Interaction: While the UI of 2D applications uses responsive rendering, the UI of a game engine uses instant rendering. Because the mechanisms for realizing these two are different, applications developed with a game engine face difficulties in UI layout and fluency, and complex UI effects are difficult to implement.

[0028] (3) Interaction problems with 2D application software development kits (SDKs): Due to the complexity of its configuration, the game engine cannot integrate 2D application SDKs during the development process of 3D scenes. Therefore, data communication between 3D scenes developed based on the game engine and 2D applications can only be done via an event-based method, making the implementation of the code complicated.

[0029] In view of the above, the embodiment of this application provides an XR application development system for terminal development. The development system comprises a runtime container capable of providing a runtime environment during the XR application generation process (the XR application generation process is also called the XR application development process) and execution process, a content generator for generating content, and an application generation platform for packaging the application. The runtime container is integrated into the content generator and generates application content for the XR application based on the runtime environment provided by the runtime container. Once application content is generated, the application generation platform generates an XR application containing the application content and the runtime container based on the application content and the runtime container. In the execution process of an XR application developed based on the runtime container, the runtime container is executed between the application content and the operating system, and the runtime container provides a runtime environment to the application content, so that the application content is executable in the runtime environment. By adding a runtime container between the application content and the operating system, it becomes possible to shield the device's basic information, eliminating the strong dependency between the execution of the application content and the operating system, enabling the development and execution of cross-platform and cross-device XR applications, simplifying and making XR application development easier, and solving the problems of XR application development and cross-device execution.

[0030] Furthermore, runtime container-based XR application development enables cross-platform and cross-device development and execution of XR applications. This allows terminal R&D engineers to develop not only standalone XR applications for XR devices, but also XR applications that primarily incorporate 3D scenes into 2D applications. This not only meets the demand for standalone XR applications for XR devices, but also the demand for XR applications primarily incorporating 3D scenes into 2D applications for terminal devices, thereby solving the pain points of cross-device (e.g., mobile phones, AR devices, VR devices) in the XR field and adapting to the development of XR technology.

[0031] Furthermore, the XR application development system for terminal development provided by the embodiment of this application pre-develops a runtime container capable of providing a runtime environment, and also provides a content generator and an application generation platform. For developers, it is only necessary to develop application content based on the runtime container and then package the application content and runtime container as an XR application using the application generation platform. This offers advantages such as low difficulty, synchronized R&D, and guidance from complete R&D, resulting in low demands on developers and applicability to terminal R&D engineers as well as other developers such as game engineers. In addition, the existence of the runtime container simplifies XR application development, allowing for direct consideration of various application needs without being subject to various limitations imposed by game engines. This overcomes various shortcomings and problems faced in XR application development based on the aforementioned game engines, and further enables the development of XR applications with cross-device and cross-platform features.

[0032] Furthermore, by providing a runtime environment through a runtime container, application content development becomes relatively independent. Application content mainly consists of 3D scene resource data and application logic code. Specifically, the application program code is created using a dynamic language such as JavaScript. XR applications developed in this way have characteristics such as faster iteration and higher performance.

[0033] In each embodiment of this application, the XR device is not limited to its implementation form and may be an AR device or a VR device. The AR device includes, but is not limited to, AR glasses, AR helmets, etc. The VR device includes, but is not limited to, VR glasses, VR helmets, etc. Correspondingly, the terminal device that runs the 2D application incorporating the 3D scene may be, but is not limited to, a mobile phone, tablet computer, notebook computer, smart bracelet, etc.

[0034] In this application, the development system for XR applications for terminal development provided by the embodiments of this application will be described in detail with reference to the drawings in the following embodiments.

[0035] Figure 1 is a schematic diagram of the structure of an XR application development system 100 for terminal development provided by an embodiment of this application. As shown in Figure 1, the system 100 includes a content generator 20, a runtime container 30, and an application generation platform 40.

[0036] As shown in Figure 1, the XR application development system 100 of this embodiment includes a runtime container 30. The runtime container 30 is a pre-developed software stack that runs on the operating system. This software stack, which does not belong to the operating system, is provided for XR application development and is necessary for XR application development. It functions as a runtime environment during the development and execution processes of the XR application. The runtime environment is the environment necessary to execute the developed XR application or the intermediate state program code during the development process of the XR application. For example, it includes a software engine, environment variables, and execution logic necessary to execute the XR application or the intermediate state program code. In this embodiment, the software engine is a broad concept and encompasses various engines, library functions, etc.

[0037] Based on the fact that the XR application development system 100 includes a runtime container 30, the XR application development process in this embodiment includes a process of developing the application content necessary for the XR application based on the runtime environment provided by the runtime container 30, and a process of generating the XR application based on the runtime container and the application content. The content generator 20 is for generating the application content of the XR application based on the runtime environment provided by the runtime container 30, and the runtime container 30 is integrated into the content generator 20 so that the content generator 20 can utilize the runtime environment provided by the runtime container 30. In this embodiment, by developing the application content based on the runtime environment provided by the runtime container 30, the application content has the ability to run in the runtime environment provided by the runtime container 30, and the effects of the application content during development and the effects of the application content during execution can be matched.

[0038] When application content is generated, the application generation platform 40 generates an XR application based on the runtime container 30 and the application content. In this embodiment, unlike conventional application programs, the XR application simultaneously includes the application content and a runtime container 30 for providing the runtime environment to the application content. Since the application content is developed based on the runtime environment provided by the runtime container 30, it has the capability to be executed in the runtime environment provided by the runtime container 30. Based on this, by integrating the runtime container 30 and the application content to form an XR application, during the execution process of the XR application, the runtime container 30 becomes executable between the application content and the operating system of the target device on which the XR application is located, and by providing the runtime environment to the application content, the application content is executed in the runtime environment provided by the runtime container, ultimately achieving the objective of the XR application being executed successfully. The target device on which the XR application is located may be an XR device or a terminal device running a host 2D application, but is not limited to this.

[0039] In this embodiment, the runtime container 30 runs on the operating system of the target device where the XR application is located. Providing a runtime environment for the application content of the XR application by the operating system specifically means that the operating system of the target device utilizes various software and hardware resources of the target device, such as the CPU, GPU, memory, network card, and bandwidth, and based on these software and hardware resources, provides the runtime environment necessary for the execution of the application content of the XR application. As a result, the application content is executed in the runtime environment, and 3D scenes, interactions and / or controls based on 3D scenes are displayed to the user.

[0040] In this embodiment, the implementation form of the XR application may be an independent application, or it may be an application in which a 3D scene is incorporated into a 2D application. Depending on the implementation form of the XR application, the method of generating the XR application by the application generation platform 40 will differ. The function of the application generation platform 40 is to package the runtime container 30 into a corresponding installation package and distribute it so that the target device can install the runtime container 30 locally according to the corresponding installation package, depending on the implementation form of the XR application.

[0041] If the XR application is in the form of a standalone application, the application generation platform 40 may integrate the runtime container 30 and the application content into an installation package for the XR application and distribute the installation package for the XR application. This makes the XR application downloadable, installable, and runable, just like any other standalone application. Specifically, the XR application may be an AR application run on an AR device, or a VR application run on a VR device, but is not limited to these.

[0042] If an XR application is primarily a 2D application with a 3D scene embedded within it, the 2D application that requires the 3D scene to be embedded is called the host 2D application. The application generation platform 40 may integrate the runtime container 30 into the host 2D application's installation package and distribute the host 2D application's installation package externally. In this way, when the host 2D application is downloaded, installed, and executed like other standalone applications, the runtime container 30 is also downloaded, installed, and executed. The application content of the XR application may be distributed independently of the host 2D application, or it may be included in the host 2D application's installation package and distributed together with the installation package, but this is not limited to these options. Selectively, considering the large amount of data in the XR application's content, the application content of the XR application may be distributed independently to the resource server corresponding to the host 2D application. Thus, when it is necessary to execute a 3D scene provided by an XR application during the execution process of a host 2D application, the application content of the XR application can be downloaded from the resource server as needed and executed in the runtime environment provided by the runtime container 30. This makes it possible to display a 3D scene embedded in a page of the host 2D application on a page of the host 2D application.

[0043] It should be noted that the embodiments of this application do not limit the host 2D application. The host 2D application may be any 2D application that requires 3D scene display, such as e-commerce applications, entertainment applications, educational applications, medical applications, etc.

[0044] The runtime container 30 in this embodiment provides a runtime environment that runs on the operating system, and the method of providing the runtime environment by the runtime container 30 is the same or similar during both the development process of the XR application and the execution process of the XR application. Figure 2 shows one implementation configuration of the runtime container 30 provided by this embodiment, which includes a framework layer 31, a runtime environment layer 32, and a software library layer 33. The three layers will be described below.

[0045] The Libraries layer 31 includes a set of foundational libraries necessary for XR applications. These foundational libraries provide various software engines and / or library functions, including, but are not limited to, a 3D rendering engine, an AI inference engine, an engine or library function provided by a download library to implement download functionality, an engine or library function provided by a storage library to implement storage functionality, an engine or library function provided by a network library to implement network functionality, an engine or library function provided by a deployment library to implement deployment functionality, a blockchain engine, an engine or library function provided by a tracking library to implement tracking functionality, a 2D UI engine, an event engine, etc. Furthermore, depending on the downloadable content, the download library may be divided into an image library, an animation library, etc. These software engines are intended to provide XR applications with some of the fundamental capabilities necessary for their execution, such as 3D scene rendering capabilities, neural network model inference capabilities, various resource download capabilities, storage capabilities, network communication capabilities, deployment capabilities, etc. Various engines and software libraries are described below as examples, but are not limited to these.

[0046] The 3D rendering engine is designed to provide capabilities such as 3D rendering, animation, and physical simulation, and may, for example, employ AceNNR, but is not limited to that.

[0047] The AI ​​inference engine is designed to provide terminal-side inference execution capabilities for deep neural network models, and may, for example, be an MNN.

[0048] A blockchain engine is designed to provide blockchain services.

[0049] The network library provides efficient network transmission capabilities between the terminal side (primarily the target device where the XR application is deployed) and the cloud side (primarily the cloud server that supports the XR application).

[0050] The download library, based on the network library, provides the ability to download various resources and resource caches from the cloud, according to the encapsulated download library on the cloud side.

[0051] The image library, based on the network library and in accordance with the image download library encapsulated on the cloud side, supports the download of images in various compression formats and provides an image caching mechanism locally.

[0052] The animation library supports animation downloads in various formats, based on the network library and encapsulated on cloud-side servers, and provides a local animation caching mechanism.

[0053] The tracking library provides a log point insertion service, compressing log data generated during the execution of the XR application on the terminal side and uploading it to the cloud.

[0054] A storage library provides file storage services. If the storage library is a related data library, it provides related storage services; if the data library is a time-series data library, it provides time-series storage services, etc.

[0055] The placement library provides various placement functions for 3D models, coordinates, events, etc., in XR applications.

[0056] In this embodiment, the fundamental capabilities provided by the various software engines described above may be encapsulated as APIs, and these fundamental capabilities may be exposed externally through these APIs. These APIs are called fundamental APIs, and may also be called second-type APIs in relation to first-type APIs. First-type APIs are APIs exposed to application developers for the purpose of creating application logic code. Second-type APIs do not have to be exposed to application developers, and first-type APIs are obtained based on the encapsulation of second-type APIs. Each function entity corresponding to a first-type API may contain one or more second-type APIs, but is not limited to this. When an application developer develops application logic code based on first-type APIs and this application logic code is executed during the execution of an XR application, a call to a first-type API is triggered first, then a call to a second-type API encapsulated in a first-type API is triggered, then a call to the corresponding software engine is triggered, and finally, the display of a 3D scene, etc., is realized through the fundamental capabilities provided by the software engine.

[0057] Framework layer 32 is designed to provide the relevant capabilities necessary for the research and development of XR applications, primarily by exposing Type 1 APIs to application developers. The concrete implementation of these Type 1 APIs is largely provided by the software engine in the Libraries layer. Depending on the application scenario, the number and types of Type 1 APIs provided by the Framework layer may also differ. Several examples of Type 1 APIs are given below, but these are not exhaustive.

[0058] (1) The 3D Scene API is for developing 3D scenes, and such APIs are mainly implemented by encapsulating the 3D rendering capabilities of the 3D rendering engine in the Libraries layer. This means that application developers do not have to deal with the 3D rendering engine, and to some extent, it can lower the difficulty for application developers to understand and develop XR applications.

[0059] The 3D Scene API includes multiple APIs, and some of them are exemplified below.

[0060] XRScene represents a single 3D scene, and this API allows for the creation of a single 3D scene.

[0061] An XRModel represents a single 3D model, and this API allows for the realization of a single 3D model in a 3D scene.

[0062] XRAnchor provides a single reference coordinate system in the 3D world, and its API enables position calibration in a 3D scene.

[0063] XREvent represents events related to a system or application, and this API allows you to respond to various events in a 3D scene.

[0064] XRConfig provides a configuration for XR applications, and its API allows for various configurations of XR applications. For example, it can configure different resource loading and execution strategies for XR applications for high-end, medium-end, and low-end devices, and further, it can configure the resolution, level, type, etc., of each 3D model or other object in a 3D scene.

[0065] (2) The lifecycle management API is for managing the lifecycle of an XR application, and includes, but is not limited to, the operation, termination, and updates of the XR application. Such an API is mainly implemented by encapsulating the lifecycle management capabilities of the lifecycle management engine in the Libraries layer.

[0066] (3) The resource management API is for managing various resources of an XR application, and includes, but is not limited to, downloading, caching, and storing resources required by the XR application. Such APIs are mainly implemented by encapsulating the download and storage capabilities provided by the download engine and storage engine in the Libraries layer.

[0067] (4) The UI Interaction API is applied to application configurations in which a 3D scene is incorporated into a 2D application, and is intended to manage UI interaction between the XR application and its host 2D application, and supports the display of pages developed in the W3C style on the front end. The UI Interaction API can be implemented as a web container for handling the display functions related to the visual UI of the host 2D application in the XR application, and is an entity container for providing the GUI of the 2D application. Alternatively, it can be implemented as a Native container, which is a virtualization container. Such an API is implemented by encapsulating the UI interaction capabilities provided by the 2D UI engine in the Libraries layer.

[0068] (5) The Event Interaction API is for managing event interactions in XR applications and includes, but is not limited to, various event actions such as clicks, long presses, zooms, rotations, and proximity. Such APIs are implemented by encapsulating the event management and response capabilities provided primarily by the event engine in the Libraries layer.

[0069] The Framework layer 32 exposes a first-type API to application developers and also has several additional logic functions. For example, it has a function to sense target events during the generation or execution process of an XR application. A target event is a call event to at least one first-type API, and according to the execution logic between at least one first-type API, it executes a function entity of at least one first-type API and provides the target API encapsulated in the function entity to the runtime environment layer 33. Here, the target API is a second-type API encapsulated within a function entity of a first-type API. Different first-type APIs can usually implement different functionalities, so the second-type APIs encapsulated within their function entities will also be different, but this is not limited to them.

[0070] The runtime environment layer 33 provides runtime virtual machines (VMs) that are compatible with the programming language used in the XR application. These runtime VMs provide a runtime environment for the XR application. Specifically, they call the corresponding software engine in the Libraries layer in response to target events sensed by the Framework layer 32, depending on the target API provided by the Framework layer 32, thereby completing the execution of intermediate state resource data during the development process of the XR application, or the execution of application content in the XR application.

[0071] The VM that conforms to the programming language used in the XR application may be the JS VM or the Python® VM. If the application logic code of the XR application is written in the JS language, the JS VM is provided; if the application logic code of the XR application is written in the Python language, the Python® VM is provided. It should be explained that the provision of a runtime environment to the XR application by these runtime VMs includes providing the software and hardware resources required for the XR application using the software and hardware resources of the target device via the target device's operating system, while also calling the underlying capabilities provided by the corresponding software engine in the Libraries layer based on these software and hardware resources to perform processing such as rendering on the 3D scene resource data, interpret and execute the application logic code, enable the XR application to utilize the various underlying capabilities in the Libraries layer, and finally display the 3D scene and the process of interaction and / or control based on the 3D scene to the user.

[0072] The JS VM primarily supports application developers (e.g., terminal R&D engineers) in creating JS application logic code, dynamically distributing it, and interpreting and executing it via the JS VM, thereby enabling dynamic updates to the logic of XR applications. Examples include 3D scene UI interfaces and interaction logic.

[0073] The Python VM primarily supports application developers (e.g., terminal R&D engineers) in creating Python application logic code, dynamically distributing it, and interpreting and executing it using the Python VM, thereby enabling dynamic updates of application code (e.g., a certain algorithmic task) created in Python. For example, in an AR scene, application logic code may be created using the Python language, and specifically, a task such as a face recognition algorithm in an AR application may be created.

[0074] It should be explained here that, in the embodiments of this application, the number of runtime VMs included in the runtime environment layer 33 is not limited, and the number of runtime VMs may be one or more. For example, a JS VM or a Python VM may exist alone, or a JS VM and a Python VM may exist simultaneously. If there are multiple runtime VMs, different runtime VMs are implemented by different dynamic programming languages ​​in order to provide a runtime environment for XR applications implemented by different dynamic programming languages.

[0075] Furthermore, if there are multiple runtime VMs, the runtime environment layer 33 further includes a VM scheduling module, which, during the execution of the XR application, schedules a target VM implemented using the same dynamic programming language as the XR application from among the multiple runtime VMs, depending on the dynamic programming language used for the XR application, thereby providing a runtime environment to the XR application by the target VM. Correspondingly, during the development of the XR application, the VM scheduling module further schedules a target VM implemented using the same dynamic programming language as the XR application development, depending on the dynamic programming language used for the development of the XR application, thereby providing a runtime environment to the intermediate state resource data during the development of the XR application by the target VM.

[0076] In this embodiment, there is a correlation between the three layers of the runtime container 30, with the Libraries layer 31 located at the bottom and the runtime environment layer 33 located between the Libraries layer 31 and the Framework layer 32, as shown in Figure 3. The Libraries layer 31 is for providing various basic capabilities, with different software engines providing different basic capabilities. For example, a 3D rendering engine provides 3D rendering capabilities, and an AI inference engine provides AI model inference capabilities based on deep neural networks. These basic capabilities are exposed by encapsulation to form a different second type of API. The second type of API is exposed to the runtime environment layer 33, while the second type of API is further encapsulated into a first type of API and exposed to application developers via the Framework layer 32. The Framework layer 32 can sense target events that trigger calls to the first type of API during the development or execution process of an XR application, such as events that trigger the execution of an XR application during the execution process of an XR application, events that load static resources of a 3D scene during the development process of an XR application, trigger events that confirm the 3D scene effect after dynamic information of a 3D scene has been added to the 3D scene, or trigger events that confirm the interaction effect after application program code has been added. These target events trigger a call to at least one first type of API, execute at least one function entity of the first type of API, and when the second type of API contained in the function entity is executed, it provides the second type of API to the runtime environment layer 33. The runtime environment layer 33 receives the second type of API provided by the Framework layer 32 during the development or execution process of an XR application, and can use the second type of API to call the basic capabilities provided by the software engine in the Libraries layer 31.For information on events that load static resources for a 3D scene, trigger events that check the 3D scene effect after dynamic information has been added to the 3D scene, and trigger events that check the interaction effect after application program code has been added, please refer to the explanations below.

[0077] In this embodiment, various fundamental capabilities in the Libraries layer can be realized using the C++ language, and C++ is a cross-platform language. By realizing various fundamental capabilities in the Libraries layer based on C++, XR applications developed based on the runtime environment provided by the runtime container will have cross-platform and cross-device characteristics.

[0078] In this embodiment, if various fundamental capabilities in the Libraries layer are implemented using the C++ language, the following methods can be adopted, but are not limited to, those for exposing these fundamental capabilities.

[0079] The first method is the native publishing method, which requires developing the application logic code for the XR application in C++, and this publishing method does not offer rapid iteration capabilities.

[0080] The second approach is a non-native exposure method, meaning that application logic code for XR applications can be written using the JavaScript or Python language, allowing for dynamic updates and iterations of the application logic while leveraging the advantages of the JavaScript or Python language.

[0081] Furthermore, when adopting a non-native exposure method, API exposure can be achieved through the following two methods, as shown in Figure 3.

[0082] (1) In the Libraries layer, the second type of API is exposed to the JS VM or Python VM in the runtime environment layer using JS Binding or Python Binding technology. The JS VM or Python VM then encapsulates the second type of API as the first type of API, and then exposes the first type of API to the Framework layer.

[0083] (2) In the Libraries layer, the second type of API is exposed to the JS VM or Python VM in the runtime environment layer 33 using JS binding or Python binding technology, the second type of API is encapsulated as the first type of API, and the first type of API is directly exposed from the Libraries layer to the Framework layer.

[0084] In the embodiments of this application, the application content of the XR application mainly includes 3D scene resource data and application logic code. 3D scene resource data refers to resource data for displaying a 3D scene, and includes, for example, static resources and dynamic information of the 3D scene. Static resources of the 3D scene refer to static resources in the 3D scene, such as various 3D models and images, while dynamic information of the 3D scene refers to dynamic information related to 3D scene effects, such as the movement trajectory of a 3D model, the camera system in the 3D scene, the water system in the 3D scene, etc. This dynamic information can make the 3D scene effects richer and more diverse. Application logic code data includes various code data that reflect the application logic, and this application logic includes logic that controls and interacts with various resources in the 3D scene at the application layer, such as movement control for character models, interaction control for character models and object models, and special effects that appear when certain conditions are met.

[0085] In selectable embodiments, static resources of a 3D scene are generated in advance, and application content for an XR application is obtained by editing the dynamic information of the 3D scene and application logic code within these static resources. Based on this, the application content generation method by the content generator 20 includes loading static resources of a 3D scene in response to an XR application generation operation, and obtaining application content for an XR application by adding dynamic information of the 3D scene and application logic code to the static 3D scene resources based on the runtime environment provided by the runtime container 30.

[0086] The application logic code may be written by the application developer based on a first type of API provided to the application developer by the Framework layer in the runtime container 30.

[0087] In an optional embodiment, as shown in Figure 1, the XR application development system 100 provided by this embodiment further includes a 3D model development tool 10, which is intended for model developers and allows them to develop static resources of 3D scenes necessary for generating an XR application. Specifically, the 3D model development tool 10 can generate various 3D models in response to the model developer's 3D model development operations and output these 3D models as static resources of at least some 3D scenes.

[0088] 3D scene static resources include various static resources necessary for building 3D scenes in XR applications, including, but not limited to, the following:

[0089] (1) A 3D model file is used to describe a 3D model. Depending on the application's needs, 3D models can be classified, and in different application scenarios, 3D models may be classified in different ways. Taking the XR scene shown in Figure 5 as an example, the 3D models may be divided into three broad categories: characters, objects, and scenes, but are not limited to these. In other words, the characters, shoes, bag, etc. shown in Figure 5 all belong to the 3D model, and the entire 3D scene is also a single 3D model. In this embodiment, the format of the 3D model is not limited; for example, it may be FBX, OBJ, GItf, etc., but is not limited to these.

[0090] (2) Image resources are images that need to be displayed at certain locations in a 3D scene.

[0091] (3) Animation resources are animation information that needs to be displayed at certain locations in a 3D scene.

[0092] (4) Audio resources are audio information that needs to be played at certain locations in a 3D scene.

[0093] It should be explained that while the 3D model file mentioned above is a required resource, image resources, animation resources, and audio resources are optional resources and should be determined specifically according to the application scenario.

[0094] In this embodiment, the implementation method of the 3D model development tool 10 is not limited. Any development tool capable of providing 3D model code editing functions, code compilation functions, debugging functions, and a graphical user interface is applicable to the embodiment of this application. For example, a plugin may be created using Android Studio or Xcode, or a custom-developed IDE for XR application development may be adopted. The custom-developed IDE must have functions such as 3D model code editing functions, code compilation functions, debugging functions, and a graphical user interface.

[0095] Once the static resources of the 3D scene necessary for XR application development are obtained, the content generator 20 adds dynamic information of the 3D scene and application logic code to these static resources, ultimately obtaining the application content of the XR application. The application content of the XR application is also called the resource package of the XR application.

[0096] In this embodiment, as shown in Figure 1, one input to the content generator 20 is the static resources of the 3D scene. Based on the runtime environment provided by the runtime container 30, the dynamic information of the 3D scene is added to the static resources of the 3D scene to obtain 3D scene resource data that includes both the static resources and the dynamic information of the 3D scene. Furthermore, as shown in Figure 1, another input to the content generator 20 is application logic code. Based on the runtime environment provided by the runtime container 30, the application logic code is added to the 3D scene resource data to obtain the application content of the XR application.

[0097] In this embodiment, the internal implementation configuration of the content generator 20 is not limited. As shown in Figure 6d, one implementation structure of the content generator 20 includes a 3D scene effect editing module 21, a 3D application logic editing module 22, a content output module 23, and a runtime container 30. In a layer-level configuration, the 3D scene effect editing module 21 and the 3D application logic editing module 22 are located on top of the runtime container 30 and require a runtime environment provided by the runtime container 30.

[0098] The 3D scene effect editing module 21 loads static resources of the 3D scene in response to the generation operation of the XR application and obtains resource data of the 3D scene by editing the dynamic information of the 3D scene for the static resources of the 3D scene based on the runtime environment provided by the runtime container 30.

[0099] The 3D application logic editing module 22 is communicatively connected to the 3D scene effect editing module 21, loads resource data for the 3D scene, and retrieves the application content of the XR application by editing the application logic code for the 3D scene resource data based on the runtime environment provided by the runtime container 30.

[0100] The content output module 23 is communicatively connected to the 3D application logic editing module 22 and outputs the application content of the XR application to the application generation platform 40, which then generates the XR application based on the application content and the runtime container 30. The application content of the XR application includes 3D scene resource data and application logic code. The 3D scene resource data includes, but is not limited to, information such as a 3D scene configuration description file, mesh data, material data, texture data, and animation data. The 3D scene configuration description file includes, but is not limited to, the 3D scene configuration, lighting settings, rendering settings, camera movement, and physical collisions. The material data includes, but is not limited to, information such as glass material and metal material. The application logic code data is created in the JS language and may be represented as xxx.js, but is not limited to that.

[0101] In the process of generating application content, the runtime container 30 provides a runtime environment to the intermediate state data generated during the XR application generation process, displays execution effects on the intermediate state data, determines whether adjustments to the dynamic information of the 3D scene or the application logic code are necessary based on the execution effects, and finally obtains application content that satisfies the requirements of the execution effects.

[0102] Specifically, the 3D scene effect editing module 21 operates the runtime container 30 in response to a static resource loading operation for the 3D scene, obtains a 3D static image by executing the static resource of the 3D scene in the runtime environment provided by the runtime container 30, obtains resource data for the first intermediate state by editing the dynamic information of the 3D scene for the static resource of the 3D scene in response to an editing operation on the 3D static image, calls the runtime container 30 again in response to a scene effect confirmation operation, obtains a 3D image for the first intermediate state by executing the resource data for the first intermediate state in the runtime environment provided by the runtime container 30, and if the 3D image for the first intermediate state does not satisfy the scene effect, adjusts the dynamic information of the 3D scene until the scene effect is satisfied, and obtains resource data for the 3D scene.

[0103] Selectively, the 3D scene effect editing module 21, in response to a project creation operation, creates a project and names it, for example, 3D Scene, or AAA as the project name; in response to a static resource import operation for the 3D scene, imports static resources for the 3D scene, which include, but are not limited to, various 3D models; runs the runtime container 30, executes the static resources for the 3D scene in the runtime environment provided by the runtime container 30, and displays a 3D static image including these 3D models in the interaction interface. In this embodiment, the content generator provides an interaction interface, which is a 3D interface and has three coordinate axes: x, y, and z. Preferably, the 3D scene itself is a 3D model, and when displaying the 3D scene, the center of the 3D scene may be edited to the origin position of the xz plane of the interface, and the y value of the ground may be set to 0. Furthermore, based on this, the 3D scene effect editing module 21 edits dynamic information for these 3D models and finally obtains resource data for the target 3D scene that satisfies the request.

[0104] In this embodiment, the content generator 20 provides an interaction interface and can also display a 3D image of a first intermediate state. Furthermore, editing dynamic information of a 3D scene for static resources of a 3D scene in response to editing operations on a static 3D image includes displaying an editing widget on the static 3D image, displaying a dynamic information editing interface that shows various editable information items in response to a trigger operation on the editing widget, and acquiring the placement information of these information items as dynamic information of the 3D scene in response to placement operations on these information items, and forming resource data for the 3D scene by overlaying this dynamic information of the 3D scene with the static resources of the 3D scene.

[0105] Depending on the application of the XR application, the interface form of the dynamic information editing interface and the information items displayed may differ, but are not limited to these. An example of an information item is given. An information item may include, for example, an information item for arranging trigger actions. The configurable trigger methods for this information item include, but are not limited to, slide triggers, long-press triggers, drag triggers, and multi-click triggers. An information item may also include, for example, an information item for arranging display effects for a 3D model. The configurable display methods for this information item include, but are not limited to, animations, sound effects, and meeting participation. Furthermore, an information item may include, for example, an information item for arranging the ending form of a 3D model. The configurable ending forms for this information item include, but are not limited to, hidden at the end, off at the end, and repeat playback. In addition, depending on the application's use, related dynamic information such as camera control, lighting control, water control, scene placeholders, and pathfinding systems may be edited for static resources of the 3D scene.

[0106] The following provides an illustrative example of the process of editing the dynamic information of the 3D scene mentioned above, using static resources for the 3D scene.

[0107] (1) Camera control provides rich camera control capabilities to a 3D model in a 3D scene by editing the relevant camera control information for the 3D model in response to editing operations on the camera control of the 3D model. Editing the relevant camera control information for a 3D model includes, but is not limited to, control information for controlling the 3D model to switch between different viewpoints, control information for controlling the 3D model to move along a predetermined trajectory by the camera, and control information for controlling the 3D model to move in accordance with other 3D models or other scene objects. Thus, when displaying a 3D scene, if this camera control information is executed, the 3D rendering engine is invoked to execute this camera control information, and the 3D model can be controlled to switch between different viewpoints, move along a predetermined trajectory, or move in accordance with other 3D models or other scene objects.

[0108] (2) Regarding lighting control, the 3D rendering engine provided by the runtime container 30 supports two types of lighting: real-time lighting and bake lighting. Real-time lighting supports directional lights, while bake lighting supports directional lights, point lights, spot lights, and area lights. In response to lighting control editing operations on the 3D model, relevant lighting control information is edited for the 3D model, for example, by selecting the type of light for the 3D model and editing parameters such as the light color, illumination angle, illumination range, and illumination intensity, a lighting effect that satisfies the requirements for the 3D model is rendered in the 3D scene.

[0109] (3) Regarding water control, the 3D rendering engine provided by the runtime container 30 supports water materials, and by using water materials on a planar mesh, water surface rendering effects can be quickly realized. Water materials can be used by selecting Shader Graphs or Water Shader in the Material option. In this embodiment, parameters related to water materials include, but are not limited to, Deep Color: color of deep water, Shallow Color: color of shallow water, Strength: currently disabled, Smoothness: smoothness, Displacement: wave movement range, Normal Steength: wave normal strength, and Normal Tiling: tiling coefficient of the wave normal map. In response to water control editing operations on the 3D model, relevant water control information is edited for the 3D model, for example, by selecting a water shader for the 3D model and editing parameters such as water color, smoothness, wave movement range, and wave normal strength, a water surface rendering effect that satisfies the requirements for the 3D model is rendered in the 3D scene.

[0110] (4) Regarding scene placeholder editing, in order to use it in the application layer, placeholder information, such as placeholder information for 3D models, camera placeholder information, image and animation placeholder information for other objects, must be added to the 3D scene. Specifically, the location where placeholder information is placed in the 3D scene will differ depending on the application's Product Requirements Document (PRD). For example: 1) Image placeholders are used to mark the location where an image should be placed in the 3D scene. 2) Animation placeholders are used to mark the location where an animation should be placed in the 3D scene. 3) Bonus placeholders are used to mark the location where a bonus should be displayed in the 3D scene. 4) Regarding camera placeholders, multiple camera placeholders can be defined in the 3D scene, and in this way, it will be possible to switch camera viewpoints at runtime using these camera placeholders. 5) 3D model placeholders are used to mark the position of the corresponding 3D model in a 3D scene, enabling operations such as moving, rotating, and zooming the model, and supporting subsequent updates of the 3D model. 6) Regarding clickable area placeholders, by adding a Clickable component to a 3D model in a 3D scene, it is possible to mark clickable areas other than the 3D model in the 3D scene, and to mark the positions for collision detection in the 3D scene. In response to placeholder editing operations on the 3D scene, by editing the relevant placeholder information for the 3D scene, such as the placeholder for the 3D model, image placeholder, animation placeholder, bonus placeholder, camera placeholder, and clickable area placeholder, objects such as 3D models, images, animations, and bonuses can be displayed in the appropriate positions in the 3D scene, and rendering effects such as switching the camera viewpoint can be applied to the 3D model using the camera placeholder.

[0111] (5) Regarding the pathfinding system editing, three different areas are set according to the map of the 3D scene. 1) Walkable indicates that the area is an area where a specific model object (e.g., a character model) can move, such as the ground. 2) Not Walkable indicates an obstacle (e.g., a wall) that a specific model object cannot pass through. 3) Jump indicates that the area is an area where a specific model object can jump over, such as a threshold or a ditch. In response to pathfinding editing operations on the 3D scene, the pathfinding rendering effect of a specific model object in the 3D scene is displayed by editing the relevant pathfinding information for the 3D scene, such as areas that a specific model object can pass through, areas that it cannot pass through, or areas that it can jump over in the 3D scene.

[0112] What needs to be explained here is that, in the static resource editing process of the 3D scene described above, the 3D scene effect editing module 21 in the content generator 20 operates the runtime container 30, and in the runtime environment provided by operating the runtime container 30, it can execute the resource data of the first intermediate state obtained after the addition of the dynamic information. For example, by calling the 3D rendering engine provided by the runtime container 30 and rendering the resource data of the first intermediate state, it obtains a 3D image of the first intermediate state. The dynamic information is continuously corrected based on the rendering effect of the 3D image of the first intermediate state until a rendering effect that satisfies the requirements is obtained. This dynamic information is then used as the final dynamic information, and the resource data of the 3D scene in the application content is formed by the final dynamic information and the static resources of the 3D scene.

[0113] In selectable embodiments, the dynamic information editing interface may be independent of the interface for displaying a 3D static image, or it may be implemented by being incorporated into the interface for displaying a 3D static image, but is not limited to this. Regardless of the interface form, a refresh widget or an effect confirmation widget may be placed in the dynamic information editing interface or the interface for displaying a 3D static image. The application developer can perform an operation to confirm the 3D scene effect after the dynamic information of the 3D scene has been added by clicking or long-pressing these widgets. The 3D scene effect editing module 21 responds to the scene effect confirmation operation by calling the runtime container 30 again and, in the runtime environment provided by the runtime container 30, executes the resource data of the first intermediate state to obtain a 3D image of the first intermediate state. The 3D image of the first intermediate state reflects the 3D scene effect after the dynamic information of the 3D scene has been added to the static resources of the 3D scene.

[0114] In this embodiment, the application developer may manually determine whether the 3D image of the first intermediate state satisfies the scene effect. If it is determined that the 3D image of the first intermediate state does not satisfy the scene effect, the application developer may adjust the dynamic information of the 3D scene and regenerate the resource data of the first intermediate state until a 3D image of the first intermediate state that satisfies the scene effect is obtained. The regenerated resource data of the first intermediate state may then be executed in the runtime environment provided by the runtime container 30. The 3D scene resource data is formed using the dynamic information and static resources of the 3D scene at this time. Alternatively, a functional module or component may be generated to identify and determine the scene effect for various 3D images. This functional module or component may automatically capture the screen state of the 3D image of the first intermediate state and determine whether the 3D image of the first intermediate state satisfies the scene effect according to the screen state.

[0115] Specifically, the 3D application logic editing module 22 operates the runtime container 30 in response to a 3D scene resource data loading operation, and obtains a 3D image of the first intermediate state by executing the 3D scene resource data in the runtime environment provided by the runtime container 30. In response to an application logic code editing operation, it generates resource data of the second intermediate state based on the 3D scene resource data and the application logic code, calls the runtime container 30 again, and obtains a 3D image of the second intermediate state by executing the resource data of the second intermediate state in the runtime environment provided by the runtime container 30. If the 3D image of the second intermediate state does not satisfy the interaction effect, it obtains the application content of the XR application by adjusting the application logic code until the interaction effect is satisfied.

[0116] In this embodiment, the execution effect of resource data in the second intermediate state, that is, the execution effect of application logic code in the 3D scene, is divided into two types: scene control and scene interaction.

[0117] Scene control is the control of 3D models in a 3D scene in response to user input. In other words, application logic code has the function of controlling 3D models in a 3D scene in response to user input. The method of controlling 3D models in a 3D scene is not limited; for example, it is possible to control the 3D model to move, jump, etc., or to control the 3D model to switch camera viewpoints. For example, in the scene shown in Figure 5, the character model is controlled to move in response to the user's joystick operation (an example of user input), and in response to the user's input, the new position coordinates of the character model, for example, world coordinates [0.15,0.22,0.31], are calculated, and the character model is moved from the original position coordinates to the new position coordinates. It should be noted that the user input method differs depending on the target device displaying the 3D scene. If the target device is a mobile terminal such as a cell phone, the user can input using gestures, voice, or touch. If the target device is a VR device or AR device, the user can input using controller buttons or pause. For example, touch input may be operations such as clicking, long-pressing, or double-clicking on the 3D scene interface by the user, or it may be a trigger operation on a widget in the 3D scene by the user.

[0118] Scene interaction primarily involves triggering various response events or interaction information in response to user input or 3D models manipulated by the user. In other words, application logic code has the function of responding to user input or monitoring the state of 3D models manipulated by the user, and triggering response events or interaction information in response to user input or the state of the 3D model. For example: 1) When the user clicks on a 3D model, the 3D model can be controlled to be shown or hidden, or related special effects can be displayed. 2) When the user manipulates a 3D model and approaches a certain area or object, bonus information or other interaction information can be displayed in the 3D scene.

[0119] Based on the above, if the 3D image of the second intermediate state does not satisfy the interaction effect, adjustments to the application logic code are exemplified below, but are not limited to these. If the 3D model position is not ideal, the interaction position with the 3D model, the type of trigger operation on the 3D model, etc., are corrected in the application logic code. If the bonus information displayed in the 3D scene does not meet the application's requirements, the bonus information in the application logic code is corrected to flop information, etc., until an application logic code that satisfies the interaction effect is obtained, and the application content of the XR application is formed using this application logic code and the resource data of the 3D scene.

[0120] In this embodiment, the content generator 20 provides an interaction interface, and the interaction interface can display the 3D image of the first intermediate state and the 3D image of the second intermediate state. Similarly, the method for determining whether the 3D image of the second intermediate state satisfies the interaction effect can be referenced from the method for determining whether the 3D image of the first intermediate state satisfies the scene effect described above, which will not be repeated here.

[0121] In this embodiment, the application logic code is pre-written by the application developer, and the runtime container of this embodiment, in addition to providing a runtime environment, can provide the application developer with APIs necessary for XR application development. For the sake of explanation and distinction, these APIs for application developers are referred to as Type 1 APIs, and application developers can create the application logic code necessary for XR applications based on Type 1 APIs. Type 1 APIs encapsulate Type 2 APIs, which are APIs provided externally by the software engine provided by the runtime container 30. Type 2 APIs allow the software engine provided by the runtime container to be invoked, and these software engines further provide intermediate state resource data or the runtime environment necessary for XR applications. Because Type 1 APIs encapsulate Type 2 APIs, their number is far smaller than that of Type 2 APIs, and Type 1 APIs are more concrete and easier to understand. This allows application developers to create application logic code more easily and efficiently, thereby lowering the difficulty of XR application development.

[0122] It should be explained here that, depending on the implementation configuration of the runtime container 30, the method of executing the static resources of the 3D scene, the resource data of the 3D scene, and the resource data of the first intermediate state and the resource data of the second intermediate state in the runtime environment provided by the runtime container 30 will differ. The implementation configuration of the runtime container 30 shown in Figure 2 will be explained in detail below.

[0123] In the process of editing dynamic information in a 3D scene, The 3D scene effect editing module 21 operates the runtime container 30 in response to a loading operation of static resources of the 3D scene. When the runtime container 30 operates, the Framework layer senses a loading event of static resources of the 3D scene, which is a specific example of the target event described above, and determines at least one first type API corresponding to the event, the first type API includes, but is not limited to, the 3D scene API, lifecycle management API, resource management API, etc. Then, according to the logic relationships between these APIs, the Framework layer executes the function entities of these APIs, and a second type API is encapsulated within these function entities. For example, the function entity of the 3D scene API encapsulates the second type API corresponding to the 3D rendering engine, the function entity of the lifecycle management API encapsulates the second type API corresponding to the lifecycle management engine, and the function entity of the resource management API encapsulates the second type API corresponding to the resource management engine. The second type of API encapsulated within these function entities is called a target API. When any target API is executed, the Framework layer provides the target API to the runtime environment layer, which then calls the corresponding software engine depending on the target API and responds to events that load static resources for the 3D scene. For example, the runtime environment layer obtains a 3D static image by calling a 3D rendering engine to render the static resources for the 3D scene, and it calls a lifecycle management engine to manage the lifecycle of the 3D static image, for example, by displaying or turning it off. It also calls a resource management engine to perform resource management for the static resources of the 3D scene, such as local storage and caching.

[0124] The 3D scene effect editing module 21, in response to the scene effect confirmation trigger operation, invokes the runtime container 30 again. When the runtime container 30 is invoked again, the Framework layer senses an event that confirms the 3D scene effect after dynamic information of the 3D scene has been added, and this event is a concrete example of the target event described above. The Framework layer determines at least one first type API corresponding to this event, and the first type API includes, but is not limited to, the 3D scene API, lifecycle management API, resource management API, etc. Based on the logic relationships between these APIs, the Framework layer executes the function entities of these APIs, and a second type API is encapsulated within these function entities. For example, the 3D scene API function entity encapsulates the second type API corresponding to the 3D rendering engine, the lifecycle management API function entity encapsulates the second type API corresponding to the lifecycle management engine, and the resource management API function entity encapsulates the second type API corresponding to the resource management engine. The second type of API encapsulated within these function entities is called a target API. When any target API is executed, the Framework layer provides the target API to the runtime environment layer, which then calls the corresponding software engine depending on the target API and responds to events that load static resources for the 3D scene. For example, the runtime environment layer obtains a 3D image of the first intermediate state by calling a 3D rendering engine to render the resource data of the first intermediate state. It calls a lifecycle management engine to manage the lifecycle of the 3D image of the first intermediate state, for example, by displaying or turning it off. It calls a resource management engine to perform resource management on the resource data of the first intermediate state, for example, by local storage or caching.

[0125] During the editing process of application logic code, The 3D application logic editing module 22 operates the runtime container 30 in response to a 3D scene resource data loading operation. When the runtime container 30 operates, the Framework layer senses a 3D scene resource data loading event, which is a specific example of the target event described above, and determines at least one first type API corresponding to the event, the first type API includes, but is not limited to, the 3D scene API, lifecycle management API, resource management API, UI interaction API, and event interaction API. Based on the logical relationships between these APIs, the Framework layer executes the function entities of these APIs, encapsulating a second type of API within each of these API function entities. For example, the 3D Scene API function entity encapsulates a second type of API corresponding to the 3D rendering engine, the Lifecycle Management API function entity encapsulates a second type of API corresponding to the Lifecycle Management engine, the Resource Management API function entity encapsulates a second type of API corresponding to the Resource Management engine, the UI Interaction API function entity encapsulates a second type of API corresponding to the 2D UI engine, and the Event Interaction API function entity encapsulates a second type of API corresponding to the Event Management engine. These second type of APIs encapsulated within function entities are called target APIs. When any of these target APIs are executed, the Framework layer provides the target API to the Runtime Environment layer, which, based on the target API, invokes the corresponding software engine to respond to the loading event of static resources for the 3D scene. Exemplarily, the Runtime Environment layer obtains a 3D image of the first intermediate state by invoking the 3D rendering engine to render the resource data for the 3D scene. The lifecycle management engine is invoked to manage the lifecycle of the 3D image in the first intermediate state, for example, by displaying or turning it off. The resource management engine is invoked to manage the resource data of the 3D scene, for example, by performing resource management such as local storage and caching.Furthermore, the event engine responds to interaction events during the execution of resource data in the 3D scene, such as long presses, clicks, and rotations in the interface. If an XR application is implemented in a form where a 3D scene is embedded in a host 2D application, the application logic code may include functionality to perform UI interactions with the host 2D application, in which case the 2D UI engine can respond to and react to UI interactions between the application logic code and the host 2D application.

[0126] The 3D application logic editing module invokes the runtime container 30 again in response to an editing operation on the application logic code. When the runtime container 30 is invoked again, the Framework layer senses an editing event in the application logic code, which is a specific example of the target event described above, and determines at least one first type API corresponding to the event, which includes, but is not limited to, the 3D scene API, lifecycle management API, resource management API, UI interaction API, and event interaction API. Based on the logic relationships between these APIs, it executes the function entities of these APIs, which encapsulate a second type API. The second type API encapsulated in these function entities is called the target API, and when any target API is executed, the Framework layer provides the target API to the runtime environment layer, which invokes the corresponding software engine according to the target API and responds to an event to load static resources of the 3D scene. A detailed explanation can be found in the above embodiment and will not be repeated here.

[0127] Based on the above operations, application content for an XR application can be generated. Subsequently, the application generation platform 40 packages the application content and runtime container to generate the XR application. Depending on the application form of the XR application, it may be deployed to the target device using a different method. Once the XR application is deployed to the target device, the XR application can be executed on the target device. Regardless of the method, the execution process of the XR application involves running the runtime container 30 in the XR application. The runtime container 30 is executed between the application content of the XR application and the operating system of the target device on which the XR application is deployed. The runtime container 30 provides a runtime environment to the application content of the XR application, executes the application content in the runtime environment, and displays the 3D scene and the process of interaction or control within the 3D scene to the user. As shown in Figure 4, the runtime framework of the XR application has the application content of the XR application at the top, the runtime container 30 in the middle, and the operating system of the target device on which the XR application is installed at the bottom. The runtime container 30 runs on the operating system of the target device and provides a runtime environment for the application content of the XR application. The application content of the XR application runs in this runtime environment, displays a 3D scene to the user, and performs interactions and / or controls based on the 3D scene. Here, interactions include user-triggered interactions and interactions and / or controls that are automatically triggered within the 3D scene according to the application's needs.

[0128] In the above framework, the Framework layer 32 of the runtime container 30 senses a target event during the execution process of the XR application, executes a function entity of at least one first type API corresponding to the target event, and provides the target API encapsulated in the function entity to the runtime environment layer 33. The target API is a second type API encapsulated in the function entity. Based on the target API provided by the Framework layer 32, the runtime environment layer 33 responds to the target event by calling the corresponding software engine in the Libraries layer 31. Specifically, the runtime environment layer 33 responds to the target event by calling the corresponding software engine in the Libraries layer 31 based on the target API, and can communicate the response result to the application interface of the XR application via the Framework layer 32.

[0129] During the execution of an XR application, target events that the Framework layer 32 can sense include, but are not limited to, trigger events for executing the XR application, interaction events with target objects in the XR application, and trigger events for turning off the XR application. Target objects in the XR application include, but are not limited to, 3D models in a 3D scene, 2D objects, and at least one in an interactable region. As shown in Figure 5, a 3D model may be a character model, an item model, or an entire 3D scene model, and correspondingly, a 2D object includes, but is not limited to, images, sounds, or animations that need to be displayed in a 3D scene.

[0130] In selectable embodiments, the XR application is implemented as an independent application, and the user can trigger the execution of the XR application by clicking, double-clicking, or long-pressing the XR application icon, and the Framework layer 32 can generate a trigger event to execute the XR application in response to the trigger operation of the XR application icon.

[0131] In another selectable embodiment, the XR application is implemented as an application form in which a 3D scene is embedded in a host 2D application, in which case the host 2D application is provided with an access portal to the 3D scene, and the user can perform a trigger operation to run the XR application by clicking, double-clicking, or long-pressing the access portal, and the Framework layer 32 can generate a trigger event to run the XR application in response to the trigger operation of the access portal of the XR application.

[0132] Depending on the application scenario of the XR application, the first type of API that responds to trigger events for executing the XR application may differ, but this is not limited to the embodiments of this application. For example, the first type of API that responds to trigger events for executing the XR application includes a 3D scene API, a lifecycle management API, and a resource management API, and the function entities of these APIs encapsulate the second type of API corresponding to the software engine. For example, the function entity of the 3D scene API encapsulates the second type of API corresponding to the 3D rendering engine, the function entity of the lifecycle management API encapsulates the second type of API corresponding to the lifecycle management engine, and the function entity of the resource management API encapsulates the second type of API corresponding to the resource management engine. The Framework layer 32 sequentially executes the function entities of the first type of API according to the application logic, and when the second type of API encapsulated in these function entities is executed, it provides the second type of API to the runtime environment layer 33 as the target API. Based on the target API, the runtime environment layer 33 responds to the trigger event for calling the corresponding software engine to execute the XR application. For example, the runtime environment layer 33 invokes the 3D rendering engine, invokes the lifecycle management engine to download the resource package of the XR application, invokes the 3D rendering engine to render the downloaded resources, and the runtime environment layer feeds back the download and resource load status to the Framework layer 32, thereby completing the initialization of the 3D scene of the XR application. Here, in the process, the resource management engine may also be invoked to manage the resource package of the XR application, for example, local storage, caching, etc.

[0133] Depending on the application scenario of the XR application, the first type of API corresponding to interaction events with target objects in the XR application may differ, but this is not limited to the embodiments of this application. For example, at least one first type of API corresponding to interaction events includes, but is not limited to, a 3D scene API, a UI interaction API, and an event interaction API. A second type of API is encapsulated within the function entities of these APIs; for example, the function entity of the 3D scene API encapsulates a second type of API corresponding to a 3D rendering engine, the function entity of the UI interaction API encapsulates a second type of API corresponding to a 2D UI engine, and the function entity of the event interaction API encapsulates a second type of API corresponding to an event management engine. The Framework layer 32 sequentially executes the function entities of the first type of API according to the application logic, and when the second type of API encapsulated in these function entities is executed, it provides the second type of API to the runtime environment layer 33 as the target API. The runtime environment layer 33 responds to interaction events by calling the corresponding software engine based on the target API. For example, the runtime environment layer calls the 3D rendering engine to render resource data related to the interaction event, calls the event engine to respond to the interaction event, for example, to interaction events such as long press, click, and rotate in the interface, and if the interaction event includes UI interaction with the host 2D application, it further calls the 2D UI engine to respond to the UI interaction with the host 2D application.

[0134] Depending on the target API, the runtime environment layer 33 responds to target events by calling the corresponding software engine in the Libraries layer 31 in a different manner. The following examples illustrate this using some of the software engines included in the Libraries layer 31.

[0135] If the target API includes a second type of API corresponding to the download engine in Libraries layer 31, the download engine is invoked to download the application content of the XR application from the server.

[0136] If the target API includes a second type of API corresponding to the memory engine in Libraries layer 31, the memory engine is invoked to perform memory management for the application content of the XR application.

[0137] If the target API includes a second type of API corresponding to the lifecycle management engine in Libraries layer 31, the lifecycle management engine is invoked to perform lifecycle management for the XR application.

[0138] If the target API includes a second type of API corresponding to the 3D rendering engine in Libraries layer 31, the 3D rendering engine is invoked to perform 3D rendering for the XR application.

[0139] If the target API includes a second type of API corresponding to the AI ​​inference engine in Libraries layer 31, the AI ​​inference engine is invoked to perform AI inference processing on the XR application.

[0140] If the target API includes a second type of API corresponding to the 2D UI engine in Libraries layer 31, the 2D UI engine is invoked to handle UI interactions between the XR application and the host 2D application.

[0141] If the target API includes a second type of API corresponding to the event engine in Libraries layer 31, the event engine is invoked to process the event callback logic in the XR application.

[0142] In the following embodiment, we will take the example of an application in which an XR application is an application in which a 3D scene is embedded in a host 2D application. If the host 2D application is an e-commerce app, then in order to improve the interest and user experience of the e-commerce app, a 3D scene may be embedded in the e-commerce app, for example, an AR try-on scene or a VR shopping scene. Using this scenario as an example, we will explain the execution process of the runtime container provided by the embodiment of this application.

[0143] As shown in Figure 5, an access portal for the XR application is placed on a page of the e-commerce app, and the user clicks on the access portal to enter the 3D scene space displayed in the XR application. In this embodiment, the application content (also called the resource package) of the XR application includes 3D scene resource data and application logic code created by JS, and was developed based on the above development system. During the execution of the e-commerce app, when the user triggers access to the 3D scene space of the XR application, a three-tier configuration in the runtime container is invoked, and this three-tier configuration can support the display and interaction of the 3D scene space, including, but not limited to, the following:

[0144] 1. Managing the XR application lifecycle The runtime container, in response to trigger operations of the XR application, calls the API corresponding to the download library from the second type of API to download the resource package of the XR application and complete the resource loading of the XR application. It also calls the API corresponding to the 3D rendering engine from the second type of API to render the XR application as a 3D scene using the 3D rendering engine, thereby displaying the 3D scene space on the relevant page of the e-commerce app. Furthermore, in response to logout or off operations of the XR application, it performs actions such as cleaning up related resources (e.g., memory, CPU) when the XR application is logged out, thereby realizing lifecycle management for the XR application.

[0145] Furthermore, the management process of the XR application lifecycle will be explained with reference to the three-tier configuration of the runtime container. As shown in Figure 6a, the initialization process for an XR application includes the framework layer entering the 3D scene in response to a trigger operation that enters the 3D scene space, triggering the initialization of the 3D scene by the runtime VM, the runtime VM calling the download library in the software library layer to download the resource package of the XR application, the runtime VM calling the 3D rendering engine to render the 3D scene according to the downloaded resources, updating the resource download and rendering status, and providing the resource download and rendering status to the framework layer, the download library notifying the runtime VM that the download of the resource package is complete when the download of the XR application's resource package is complete, the runtime VM notifying the framework layer that the initialization of the 3D scene is complete, and the framework layer completing the UI operations in the 3D scene. It should be noted that the resource package of the XR application may be encrypted and compressed. In this case, before rendering, the runtime VM may further call the corresponding engine to decrypt and decompress the resource package.

[0146] 2. The user triggers a first type of operation in the 3D scene space. A first type of operation is an operation that triggers a function call, and the called function responds to a first type of operation in response to the first type of operation.

[0147] Using the 3D scene space shown in Figure 5 as an example, the first type of operation triggered by the user includes, but is not limited to, the following:

[0148] The user moves a character model by manipulating a joystick in a 3D scene. This operation triggers a series of actions, such as changes in position and scene while roaming, which can be responded to and processed by callback functions.

[0149] The user controls the camera viewpoint. This control triggers a change in the scene, which can be responded to and processed by a callback function.

[0150] The user clicks the screen to trigger the playback function. This action triggers the playback of music or special effects, the switching of maps, etc., and these changes can be responded to and processed by a callback function.

[0151] Furthermore, the three-tiered architecture of the runtime container is used to describe the process by which a user triggers a function call. As shown in Figure 6b, the process of a first type of user-triggered operation includes the following:

[0152] Case 1: The framework layer responds to a user's movement operation of the character model by notifying the runtime VM of the user's movement operation of the character model, and the runtime VM invokes the 3D rendering engine in the software library layer to re-render the character model's position.

[0153] Case 2: The framework layer responds to a user's camera viewpoint switching operation by notifying the runtime VM of the user's camera viewpoint switching operation. The runtime VM then calls the 3D rendering engine in the software library layer to re-render the 3D scene or 3D model, thereby obtaining the 3D scene or 3D model from the new viewpoint.

[0154] Case 3: The framework layer responds to the user's operation to play music and special effects by notifying the runtime VM of the user's operation to play music and special effects. The runtime VM then calls the multimedia library and 3D special effects library in the software library layer to play the music and special effects.

[0155] 3. The user triggers a second type of operation in the 3D scene space. A second type of operation is an operation that triggers an event callback, and the called event responds to a second type of operation in response to the second type of operation.

[0156] Taking the case shown in Figure 5 as an example, triggering an event callback by a user performing a second type of operation includes, but is not limited to, the following: When the user manipulates and moves a character model, and the 3D rendering engine performs collision detection and determines that the character model is approaching or moving away from a predetermined model object, an event callback is triggered to cause a response from the upper layer application logic code layer, for example, to pop up a bonus, coupon, or event message.

[0157] Furthermore, the process by which a user triggers an event callback will be explained by referring to the three-tiered architecture of the runtime container. As shown in Figure 6c, the process by which a user triggers a second type of operation includes the following:

[0158] Case 1: The VM invokes the 3D rendering engine in the software library layer to monitor whether a 3D model (e.g., a character model) has collided with a given object, a process abbreviated as physical collision detection. If it detects that the character model is close to a predefined object, it provides feedback to the framework layer via the runtime VM, which then responds by executing the corresponding code data in the target application logic code data, for example, displaying bonus information.

[0159] Case 2: The VM invokes the 3D rendering engine in the software library layer to monitor whether a 3D model (e.g., a character model) has collided with a given object, a process known as physical collision detection. If it detects that the character model has moved away from a predefined object, it feeds this back to the framework layer via the runtime VM, which responds by invoking the corresponding code data in the target application logic code data, for example, by canceling the display of bonus information.

[0160] The XR application development system provided by the embodiments of this application has the following beneficial effects:

[0161] (1) Cross-device and high performance: Basic capabilities in the Libraries layer are realized using C++, and optimal performance is obtained through precise optimization. For example, parallel computing acceleration is performed on the command set of the device to realize cross-device at the code execution level, and furthermore, by encapsulating these basic capabilities and obtaining APIs, secondary encapsulation can be performed for APIs of different devices. As a result, XR applications can also perform cross-device at the interaction level, further improving the performance of XR applications.

[0162] (2) High-speed iteration: A dynamic programming language such as Python or JS is selected to implement the application logic code and runtime VM of the XR application. The runtime VM is used as a glue layer to call the underlying capabilities of the Libraries layer implemented by C++ at the lowest level. The iteration frequency of the underlying capabilities of the Libraries layer is low, and iteration is possible through version updates. However, the application logic of the XR application is updated frequently, and when implemented with a dynamic programming language, the advantages of the dynamic programming language are beneficial for the rapid distribution and updating of the application logic, while simultaneously ensuring global high performance.

[0163] (3) Lightweight: Lightweight processing is performed on the software engine or software libraries in the Libraries layer according to the development needs of the XR application. For example, some unnecessary software libraries are discarded, and some software libraries are introduced using a grafting method rather than direct porting. Furthermore, the software engine or software libraries in the Libraries layer are distributed using a dynamic distribution method rather than being distributed together with the installation package of the XR application or the host 2D application, thereby achieving further weight reduction. If an AI inference engine is included, an MNN engine is adopted, and operations such as image processing and matrix calculations are performed based on the MNN, which is advantageous in reducing the package size of the AI ​​inference engine on the terminal side, and thus achieving further weight reduction.

[0164] (4) Low difficulty: By encapsulating the fundamental capabilities in the Libraries layer and exposing a first-type API to application developers via the Framework layer, it abstractly encapsulates the fundamental capabilities such as rendering, physics, and animation provided by the 3D rendering engine, and provides fewer concepts such as XRScene, XRModel, XRAnchor, XREvent, and XRConfig, making it easy for XR application developers to understand and use.

[0165] (5) As the XR application is executed, the stability of the XR application is monitored, and various data such as crash information, operational performance, and page performance of the XR application can be obtained in real time. Based on this monitoring data, subsequent operational maintenance operations are performed on the XR application, and these operational maintenance operations include, but are not limited to, patch repair and information deployment.

[0166] In this embodiment, an editing toolchain, i.e., a content generator 20, is constructed to be applied when a host 2D application 3D scene is incorporated based on a runtime container. Its structure and function can be described by referring to the embodiment shown in Figure 6d above, so it will not be repeated here. Based on the runtime container, the content generator 20 can satisfy constraints such as package size, performance, and effects when incorporated into a 2D application, and has the following effects.

[0167] (1) Improved efficiency: Significantly reduces communication costs related to art and technology. By integrating a runtime container into the content generator, the 3D rendering engine in the runtime container's software library layer enables WYSIWYG editing of 3D scene effects, making it easier for application developers to edit 3D scene effects and application logic code.

[0168] (2) Effect consistency: By having the 3D content generator and the target device share the same runtime container, consistency between content editing and the effects executed on the target device can be achieved.

[0169] (3) Standard constraints: The 3D content generator incorporates custom 3D model standards for geometry, materials, lighting, cameras, animations, etc. The export of the final 3D scene is validated according to the 3D model standards to ensure the standardization of the final 3D model and to meet constraints on effects, performance, etc. when run on target devices of the XR application.

[0170] In addition to providing an XR application development system, embodiments of this application further provide a method for developing an XR application, a method for editing an XR application, and a method for executing an XR application. The following embodiments will describe the methods for developing an XR application, editing an XR application, and executing an XR application, respectively.

[0171] Figure 7a is a schematic flowchart of a method for developing an XR application provided by an embodiment of the present application. The method is applied to the above-mentioned XR application development system and, as shown in Figure 7a, includes the following steps.

[0172] In step 71a, a runtime container necessary for generating an XR application is developed in advance to provide a runtime environment during the generation and execution processes of the XR application, and this runtime container is integrated into the content generator.

[0173] In step 72a, the content generator, which integrates the runtime container, generates application content for the XR application based on the runtime environment provided by the runtime container.

[0174] In step 73a, an XR application is generated based on the application content and runtime container. During the execution of the XR application, the runtime container is executed between the application content and the operating system of the target device on which the XR application is located, and the application content is executed in the runtime environment provided by the runtime container.

[0175] In an optional embodiment, in a content generator with an integrated runtime container, the step of generating application content for the XR application based on a runtime environment provided by the runtime container includes: loading static resources of a 3D scene in response to an XR application generation operation; obtaining resource data for a 3D scene by adding dynamic information of the 3D scene to the static resources of the 3D scene based on the runtime environment provided by the runtime container; and obtaining application content for the XR application by editing application logic code for the resource data of the 3D scene based on the runtime environment provided by the runtime container. Detailed embodiments of each step can be found in the relevant descriptions of the system embodiment described above or the method embodiment described below, and will not be repeated here.

[0176] In an optional embodiment, the step of generating the XR application based on the application content and the runtime container includes the steps of: integrating the runtime container and the application content into an installation package for the XR application and distributing the installation package for the XR application; and / or integrating the runtime container into an installation package for a host 2D application and distributing the installation package for the host 2D application; and distributing the application content to a resource server corresponding to the host 2D application, wherein the host 2D application is a 2D application into which the XR application must be incorporated.

[0177] Figure 7b is a schematic flowchart of a method for editing an XR application provided by an embodiment of the present application. The method is applied to a content generator in the development system for the above-mentioned XR application, and the content generator integrates a runtime container for providing a runtime environment during the generation or execution process of the XR application. As shown in Figure 7b, the method includes the following steps.

[0178] In step 71b, static resources of the 3D scene are loaded in response to a 3D resource editing operation.

[0179] In step 72b, resource data for the 3D scene is obtained by adding dynamic information of the 3D scene to the static resources of the 3D scene, based on the runtime environment provided by the runtime container.

[0180] In step 73b, the application content of the XR application is obtained by editing the application logic code for the resource data of the 3D scene based on the runtime environment provided by the runtime container, and the XR application is formed by the application content and the runtime container, and the runtime container in the XR application provides the runtime environment to the application content during the execution process of the XR application.

[0181] In one of the selectable embodiments, the step of obtaining 3D scene resource data by adding dynamic information of the 3D scene to the static resources of the 3D scene, based on the runtime environment provided by the runtime container, is: The steps include: obtaining a 3D static image by operating a runtime container in response to a loading operation of static resources for a 3D scene, and executing the static resources of the 3D scene in the runtime environment provided by the runtime container; Step 1: To obtain resource data of a first intermediate state by editing the dynamic information of the 3D scene for static resources of the 3D scene in response to editing operations on a 3D static image, The steps include: obtaining a 3D image of the first intermediate state by calling the runtime container again in response to a scene effect confirmation operation and executing the resource data of the first intermediate state in the runtime environment provided by the runtime container; The process includes, if the 3D image of the first intermediate state does not satisfy the scene effect, adjusting the dynamic information of the 3D scene until the scene effect is satisfied, and obtaining resource data for the 3D scene.

[0182] In one of the selectable embodiments, the step of obtaining the application content of an XR application by editing the application logic code for 3D scene resource data based on the runtime environment provided by the runtime container is: The steps include: obtaining a 3D image of a first intermediate state by operating a runtime container in response to a load operation of 3D scene resource data, and executing the 3D scene resource data in the runtime environment provided by the runtime container; The steps include: generating resource data for a second intermediate state based on the 3D scene resource data and the application logic code in response to an editing operation of the application logic code, calling the runtime container again, and executing the resource data for the second intermediate state in the runtime environment provided by the runtime container to obtain a 3D image of the second intermediate state; The process includes the step of obtaining the application content of the XR application by adjusting the application logic code until the 3D image of the second intermediate state satisfies the interaction effect, if the 3D image does not satisfy the interaction effect.

[0183] In the selectable embodiments, the runtime container includes a framework layer, a runtime environment layer, and a software library layer. The configuration and operating principles of the runtime container can be found in the embodiments described above and will not be repeated here.

[0184] Based on the above, the step of obtaining a 3D static image by executing static resources of a 3D scene in a runtime environment provided by a runtime container includes: the framework layer sensing a first load event of static resources of a 3D scene and executing a function entity of at least one first type API corresponding to the first load event; providing the first target API to the runtime environment layer when a first target API, which is a second type API encapsulated in the function entity, is executed; and the runtime environment layer responding to the first load event by calling the corresponding software engine in the software library layer according to the first target API.

[0185] Correspondingly, the step of obtaining a 3D image of the first intermediate state by executing resource data of the first intermediate state in a runtime environment provided by a runtime container includes the steps of the framework layer sensing a first edit event in the 3D static image and executing a function entity of at least one first type API corresponding to the first edit event, providing the first target API to the runtime environment layer when a first target API, which is a second type API encapsulated in the function entity, is executed, and the runtime environment layer responding to the first edit event by calling the corresponding software engine in the software library layer according to the first target API.

[0186] Correspondingly, the step of obtaining a 3D image of the first intermediate state by executing the resource data of the 3D scene in the runtime environment provided by the runtime container is: The framework layer includes the steps of sensing a second load event for resource data of a 3D scene and executing a function entity of at least one first type API corresponding to the second load event; providing the second target API, which is a second type API encapsulated in the function entity, to the runtime environment layer when the second target API is executed; and the runtime environment layer responding to the second load event by calling the corresponding software engine in the software library layer, depending on the second target API.

[0187] Correspondingly, the step of obtaining a 3D image of the second intermediate state by executing resource data of the second intermediate state in the runtime environment provided by the runtime container includes the steps of the framework layer sensing a second edit event in the application logic code and executing a function entity of at least one first type API corresponding to the second edit event, providing the second target API to the runtime environment layer when the second target API, which is a second type API encapsulated in the function entity, is executed, and the runtime environment layer responding to the second edit event by calling the corresponding software engine in the software library layer according to the second target API.

[0188] Figure 8 is a schematic flowchart of a method for executing an XR application provided by an embodiment of this application. The method is applied to a runtime container in a development system for the above-mentioned XR application, and as shown in Figure 8, the method includes the following steps.

[0189] In step 81, in response to a trigger operation to run the XR application, a runtime container in the XR application is activated, providing a runtime environment for the application content in the XR application, and the runtime container runs between the application content and the operating system of the target device where the XR application is located.

[0190] In step 82, the XR application is executed by running the application content based on the runtime environment provided by the runtime container.

[0191] In an optional embodiment, the process may further include the step of installing the runtime container and application content of the XR application on the target device according to the XR application installation package before responding to a trigger operation for running the XR application, wherein the XR application installation package includes the runtime container and application content, and correspondingly, the response to a trigger operation for running the XR application may be triggering the execution of the XR application in response to a trigger operation such as double-clicking or long-pressing the XR application icon.

[0192] and / or, In an optional embodiment, the process further includes installing a runtime container on the target device according to the host 2D application's installation package before responding to a trigger operation for running the XR application, where the host 2D application is a 2D application into which the XR application must be incorporated, the host 2D application's installation package includes the runtime container, and the application content is distributed to a resource server corresponding to the host 2D application for dynamic loading. During the execution of the host 2D application, access portal information for the XR application may be displayed, and correspondingly, the response to a trigger operation for running the XR application may be a response to a trigger operation on the access portal information, such as a click or long press. In this case, the process further includes downloading the application content from the resource server before running the application content, based on the runtime environment provided by the runtime container.

[0193] In the selectable embodiments, the runtime container includes a framework layer, a runtime environment layer, and a software library layer. The configuration and operating principles of the runtime container can be found in the embodiments described above and will not be repeated here.

[0194] Correspondingly, the step of running an XR application by executing the application content based on the runtime environment provided by the runtime container is: The framework layer senses a target event during the execution of the XR application, executes a function entity of at least one first type API corresponding to the target event, and provides the target API, which is a second type API encapsulated within the function entity, to the runtime environment layer. The runtime environment layer executes the XR application by calling the corresponding software engine in the software library layer in response to target events, depending on the target API provided by the framework layer. The first type of API is obtained by encapsulating the second type of API, and the second type of API corresponds to the software engine included in the software library layer.

[0195] The step by which the runtime environment layer responds to a target event by calling the corresponding software engine in the software library layer, depending on the target API provided by the framework layer, includes at least one of the following operations:

[0196] If the target API includes a second type of API corresponding to the download engine in the software library layer, the download engine is invoked to download the application content of the XR application from the server.

[0197] If the target API includes a second type of API corresponding to the memory engine in the software library layer, the memory engine is invoked to perform memory management for the application content of the XR application.

[0198] If the target API includes a second type of API that corresponds to the lifecycle management engine in the software library layer, the lifecycle management engine is invoked to perform lifecycle management for the XR application.

[0199] If the target API includes a second type of API corresponding to the 3D rendering engine in the software library layer, the 3D rendering engine is invoked to perform 3D rendering for the XR application.

[0200] If the target API includes a second type of API corresponding to the AI ​​inference engine in the software library layer, the AI ​​inference engine is invoked to perform AI inference processing on the XR application.

[0201] If the target API includes a second type of API corresponding to the 2D UI engine in the software library layer, the 2D UI engine is invoked to handle UI interactions between the XR application and the host 2D application.

[0202] If the target API includes a second type of API corresponding to the event engine in the software library layer, the event engine is invoked to handle the event callback logic in the XR application.

[0203] In an optional embodiment, the runtime environment layer includes at least one runtime virtual machine VM, where different runtime VMs are implemented in different dynamic programming languages ​​to provide a runtime environment for XR applications implemented in different dynamic programming languages. Correspondingly, if there are multiple at least one runtime VMs, the method further includes the step of the runtime environment layer scheduling a target VM from among the multiple runtime VMs that is implemented in the same dynamic programming language used for the XR application, so that the target VM serves the XR application, depending on the dynamic programming language used for the XR application.

[0204] In addition to the embodiments described above, embodiments of this application further provide a method for executing XR content. This method for executing XR content is applied to a host application that incorporates an XR runtime container, the XR runtime container being located in the host application's installation package and being installed together with the host application. For the implementation structure and functionality of the XR runtime container, refer to the runtime container in the embodiments described above. In this embodiment, the host application incorporates an XR runtime container for providing a runtime environment for executing XR content, and by executing the XR content in the runtime environment provided by the XR runtime container, the objective of displaying the XR content on the host application is ultimately achieved. Based on this, the method for executing XR content includes the steps of: executing a host application; displaying access portal information for XR content during the execution of the host application, wherein the access portal information may be, for example, a link or an icon, widget, etc., having a linking function on an application page provided by the host application, but is not limited thereto; and operating the XR runtime container in response to a trigger operation on the access portal information, acquiring the XR content, and executing the XR content based on the runtime environment provided by the XR runtime container. Detailed embodiments for executing XR content based on the runtime environment provided by the XR runtime container are identical or similar to the detailed embodiments for "executing application content of an XR application based on the runtime environment provided by the runtime container" in the above examples, and therefore will not be repeated here.

[0205] The embodiments of this application do not limit the method of acquiring XR content. In an optional embodiment, the XR content is located on a resource server corresponding to the host application, and the method of acquiring the XR content includes downloading the XR content from the resource server corresponding to the host application. In another optional embodiment, the XR content may be embedded in the host application, for example, if the XR content is located in the host application's installation package and can be installed along with the host application installation, the method of acquiring the XR content includes loading the XR content from a local cache.

[0206] In the embodiments of this application, the XR content may be implemented as the application content of the XR application in the above embodiment, and accordingly, the XR content and the XR runtime container can be combined to implement the XR application in the above embodiment.

[0207] Detailed embodiments and advantageous effects of each step in the above-described embodiment of the method of this application have been described in detail in the above-described embodiment, and therefore will not be described in detail here.

[0208] The implementing body for each step of the method provided by the above embodiment may be the same device, or the method may use different devices as the implementing body. For example, the implementing body for steps 71 to 73 may be device A, or for example, the implementing body for steps 71 and 72 may be device A, and the implementing body for step 73 may be device B.

[0209] Furthermore, while some of the flows described in the above embodiments and drawings include multiple operations that appear in a specific order, it is clear that these operations do not necessarily have to be executed in the order of appearance described in this document or in a synchronous manner. For example, the numbers 71, 72, etc., of the operations are simply for distinguishing each different operation, and the numbers themselves do not indicate any execution order. Also, these flows may include more or fewer operations, and these operations may be executed in order or in a synchronous manner. Note that the descriptions of "first," "second," etc., in this document are for distinguishing different messages, devices, modules, etc., and do not indicate a preceding or succeeding order, nor do they limit "first" and "second" to being of different types.

[0210] Figure 9 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application, and as shown in Figure 9, the electronic device includes a memory 1001 and a processor 1002.

[0211] Memory 1001 is for storing computer programs and is also arranged to store various other data to support operation on electronic devices. Examples of this data include commands, messages, images, animations, etc., of any application programs or methods operated on electronic devices.

[0212] The processor 1002 is coupled to the memory 1001 and executes a computer program in the memory 1001 to realize steps in the development method of an XR application, the editing method of an XR application, or the execution method of an XR application or XR content provided by the embodiment of this application.

[0213] Detailed explanations of how to develop an XR application, how to edit an XR application, how to run an XR application, or how to run XR content can be found in the above examples and will not be repeated here.

[0214] Furthermore, as shown in Figure 9, the electronic device further includes other components such as a communication component 1003, a display 1004, a power supply component 1005, and an audio component 1006. Although only some components are schematically shown in Figure 9, this does not mean that the electronic device includes only the components shown in Figure 9. The electronic device of this embodiment may be implemented as a terminal device such as a desktop computer, notebook computer, smartphone, or IoT device when executing the method for editing an XR application or the method for executing an XR application, or it may be implemented as an AR device or VR device. When the electronic device is implemented as an AR device or VR device, it may further include components such as an operation controller, but these are not shown in Figure 9. The electronic device of this embodiment may be implemented as a terminal device such as a desktop computer, notebook computer, or smartphone when executing the method for developing an XR application, on which the content generator and runtime container are executed simultaneously, or it may be implemented as a conventional server, cloud server, or a system that matches the cloud with a terminal, but is not limited to these.

[0215] Correspondingly, embodiments of the present application further provide a computer-readable storage medium in which a computer program is stored, and when the computer program is executed by a processor, the processor can implement steps in the above-mentioned method for developing an XR application or editing an XR application or steps in the method for executing an XR application or executing XR content.

[0216] The above-mentioned memory can be implemented by any type of volatile or non-volatile storage device or combination thereof, such as SRAM (Static Random-Access Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), EPROM memory (Erasable Programmable Read Only Memory), PROM (Programmable Read-Only Memory), Read-Only Memory (ROM), magnetic memory, flash memory, disks, or compact disks.

[0217] The above communication component is configured to communicate wirelessly, either wired or wirelessly, between the device on which the communication component is located and other devices. The device on which the communication component is located has access to wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G / LTE, 5G, or other mobile communication networks, or a combination thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component further includes a Near Field Communication (NFC) module for short-range communication. For example, the NFC module can be implemented based on RFID (Radio Frequency Identification) technology, IrDA (Infrared Data Association) technology, Ultra Wide Band (UWB) technology, Bluetooth® (BT) technology, and other technologies.

[0218] The above-mentioned display includes a screen, which may include a Liquid Crystal Display (LCD) and a TouchPanel (TP). If the screen includes a TouchPanel, the screen may be implemented as a touchscreen to receive input signals from the user. The TouchPanel includes one or more touch sensors for sensing touches, slides, and gestures on the TouchPanel. The touch sensors can not only sense the boundaries of touch or slide actions, but also detect the duration and pressure associated with the touch or slide operation.

[0219] The power supply component described above provides power to various components of the device in which it is located. The power supply component may also include a power management system, one or more power supplies, and other components related to the generation, management, and allocation of power in the device in which it is located.

[0220] The above audio components may be configured to output and / or input audio signals. For example, the audio component may include a microphone (MIC), and when the device on which the audio component is located is in an operating mode, such as call mode, recording mode, or voice recognition mode, the microphone is configured to receive external audio signals. The received audio signals may be stored in memory or transmitted via a communication component. In some embodiments, the audio component may further include a speaker for outputting audio signals.

[0221] As engineers in this field will know, the embodiments of this application can be provided as methods, systems, or computer program products. Accordingly, this application may take the form of complete hardware embodiments, complete software embodiments, or embodiments of a combination of software and hardware. Alternatively, this application may take the form of a computer program product implemented on one or more computer-readable storage media (including, but not limited to, disk memory, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, etc.) containing program code usable by a computer.

[0222] This application has been described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products relating to embodiments of this application. Note that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, may be realized by computer program commands. By providing these computer program commands to a general-purpose computer, a dedicated computer, an embedded processor, or the processor of another programmable data processing device, a device having functions specified by one or more flows in the flowchart and / or one or more blocks in the block diagrams can be realized by commands executed by the processor of the computer or other programmable data processing device.

[0223] These computer program commands are stored in computer-readable memory to guide a computer or other programmable data processing device to operate in a specific manner, and the commands stored in the computer-readable memory can generate a product including a command device, which implements functions specified in one or more flows of a flowchart and / or one or more blocks of a block diagram.

[0224] These computer program commands, when loaded on a computer or other programmable data processing device, generate processing to be performed on the computer by executing a series of operational steps on the computer or other programmable device, thereby providing steps for the commands executed on the computer or other programmable device to realize the functions specified in one or more flows of a flowchart and / or one or more blocks of a block diagram.

[0225] In a typical configuration, computer equipment includes one or more processors (Central Processing Units, CPUs), input / output interfaces, network interfaces, and memory.

[0226] Memory may include non-permanent memory, random access memory (RAM), and / or non-volatile memory in computer-readable media, such as read-only memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.

[0227] Computer-readable media include permanent and non-permanent, movable and immovable media, and information storage can be realized by any method or technique. Information may be computer-readable commands, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, PRAM (Phase-change Random Access Memory), SRAM, DRAM (Dynamic Random Access Memory), other types of RAM, ROM, EEPROM, flash memory, or other memory technologies, CD-ROM, DVD (Digital Video Disc), or other optical storage devices, magnetic cassettes, magnetic tape disks, or other magnetic storage devices, or any other non-transmission media capable of storing information accessible by computer equipment. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals or carriers.

[0228] Furthermore, the terms “having,” “including,” or any other variation thereof encompass non-exclusive inclusion, thereby meaning that a process, method, product, or device containing a set of elements not only contains those elements but also includes other elements not explicitly mentioned, or elements specific to such a process, method, product, or device. Unless otherwise specified, an element limited by the phrase “including one…” does not preclude the presence of other elements in a process, method, product, or device containing that element.

[0229] The foregoing is merely an example of the present application and is not intended to limit it. For engineers in the art, this application is subject to various modifications and changes. Any modifications, equivalent substitutions, improvements, etc., within the spirit and principles of this application are included within the scope of the claims.

Claims

1. A content generator that provides a runtime environment during the generation or execution process of an XR application, and which has a runtime container necessary for generating the XR application integrated internally, A 3D scene effect editing module for obtaining 3D scene resource data by loading static resources of a 3D scene in response to an XR application generation operation and editing dynamic information of the 3D scene for the static resources of the 3D scene based on the runtime environment provided by the runtime container, The system further comprises a 3D application logic editing module for editing application logic code for the resource data of the 3D scene and obtaining the application content of the XR application, based on the runtime environment provided by the runtime container, A content generator characterized in that the XR application is formed by the application content and the runtime container, and the runtime container in the XR application provides a runtime environment to the application content during the execution process of the XR application.

2. The runtime container includes a framework layer, a runtime environment layer, and a software library layer. The software library layer includes multiple types of software engines and provides a second type of application programming interface API for the multiple types of software engines, thereby obtaining the first type of API through encapsulation based on the second type of API. The framework layer provides the application developer with the first type of API so that the application developer can create the application logic code of the XR application based on the first type of API, and during the generation or execution process of the XR application, it senses a target event, executes at least one function entity of the first type of API corresponding to the target event, and provides the runtime environment layer with the target API, which is a second type of API encapsulated in the function entity. The content generator according to claim 1, characterized in that the runtime environment layer responds to the target event by calling the corresponding software engine in the software library layer according to the target API.

3. The software library layer further provides the runtime environment layer with the second type of API, obtains the first type of API through encapsulation based on the second type of API, and provides the framework layer with the first type of API so that it can provide the first type of API to application developers. or The content generator according to claim 2, wherein the software library layer further provides the second type of API to the runtime environment layer, and the runtime environment layer further provides the first type of API to the framework layer so as to obtain the first type of API by encapsulation based on the second type of API and provide the first type of API to the application developer from the framework layer.

4. The aforementioned 3D scene effect editing module specifically, In response to the loading operation of the static resources of the 3D scene, the runtime container is activated, and the static resources of the 3D scene are executed in the runtime environment provided by the runtime container, thereby obtaining a 3D static image. In response to editing operations on the 3D static image, resource data of a first intermediate state is obtained by editing the dynamic information of the 3D scene for the static resources of the 3D scene. In response to the scene effect confirmation operation, the runtime container is called again, and the resource data of the first intermediate state is executed in the runtime environment provided by the runtime container to obtain a 3D image of the first intermediate state. The content generator according to claim 2 or 3, characterized in that, if the 3D image of the first intermediate state does not satisfy the scene effect, resource data of the 3D scene is obtained by adjusting the dynamic information of the 3D scene until the scene effect is satisfied.

5. The 3D application logic editing module specifically, In response to the loading operation of the resource data of the 3D scene, the runtime container is operated, and the resource data of the 3D scene is executed in the runtime environment provided by the runtime container, thereby obtaining a 3D image of the first intermediate state. In response to the editing operation of the application logic code, resource data for a second intermediate state is generated based on the resource data of the 3D scene and the application logic code, the runtime container is called again, and the resource data for the second intermediate state is executed in the runtime environment provided by the runtime container to obtain a 3D image of the second intermediate state. The content generator according to claim 2 or 3, characterized in that, if the 3D image of the second intermediate state does not satisfy the interaction effect, the application logic code is adjusted until the interaction effect is satisfied, thereby obtaining the application content of the XR application.

6. A method for editing an XR application, applied to a content generator that integrates a runtime container for providing a runtime environment during the generation and execution processes of the XR application, The process involves loading static resources of a 3D scene in response to the XR application generation operation, The steps include obtaining resource data for a 3D scene by adding dynamic information of the 3D scene to the static resources of the 3D scene based on the runtime environment provided by the runtime container, The steps include: editing application logic code for the 3D scene resource data based on the runtime environment provided by the runtime container, and obtaining the application content of the XR application; A method for editing an XR application, characterized in that the XR application is formed by the application content and the runtime container, and the runtime container in the XR application provides a runtime environment to the application content during the execution process of the XR application.

7. The step of obtaining resource data for a 3D scene by adding dynamic information of the 3D scene to the static resources of the 3D scene, based on the runtime environment provided by the runtime container, is as follows: The steps include: operating the runtime container in response to a loading operation of the static resources of the 3D scene, and obtaining a 3D static image by executing the static resources of the 3D scene in the runtime environment provided by the runtime container; The steps include obtaining resource data of a first intermediate state by editing the dynamic information of the 3D scene for the static resources of the 3D scene in response to an editing operation on the 3D static image, The steps include: in response to a scene effect confirmation operation, calling the runtime container again and executing the resource data of the first intermediate state in the runtime environment provided by the runtime container to obtain a 3D image of the first intermediate state; The method according to 6, further comprising the step of obtaining resource data of a 3D scene by adjusting the dynamic information of the 3D scene until the scene effect is satisfied, if the 3D image of the first intermediate state does not satisfy the scene effect.

8. The runtime container includes a framework layer, a runtime environment layer, and a software library layer. The step of obtaining a 3D static image or a 3D image of the first intermediate state by executing the static resources of the 3D scene or the resource data of the first intermediate state in the runtime environment provided by the runtime container is: The framework layer senses a first load event of the static resources of the 3D scene or a first edit event in the 3D static image, executes a function entity of at least one first type of API corresponding to the first load event or the first edit event, and when a first target API, which is a second type of API encapsulated in the function entity, is executed, it provides the first target API to the runtime environment layer. The runtime environment layer includes the step of calling the corresponding software engine in the software library layer based on the first target API to respond to the first load event or the first edit event, The method according to 7, characterized in that the first type of API is obtained by encapsulating the second type of API, and the second type of API is an API corresponding to a software engine included in the software library layer.

9. The steps of editing application logic code for the 3D scene resource data and obtaining the application content of the XR application based on the runtime environment provided by the runtime container are as follows: The steps include: operating the runtime container in response to a load operation of the 3D scene's resource data, and obtaining a 3D image of a first intermediate state by executing the 3D scene's resource data in the runtime environment provided by the runtime container; The steps include: generating resource data for a second intermediate state based on the resource data of the 3D scene and the application logic code in response to an editing operation of the application logic code, calling the runtime container again, and executing the resource data for the second intermediate state in the runtime environment provided by the runtime container to obtain a 3D image of the second intermediate state; The method according to any one of claims 6 to 7, further comprising the step of obtaining the application content of the XR application by adjusting the application logic code until the interaction effect is satisfied, if the 3D image of the second intermediate state does not satisfy the interaction effect.

10. The runtime container includes a framework layer, a runtime environment layer, and a software library layer. The step of obtaining a 3D image of the first intermediate state or a 3D image of the second intermediate state by executing the resource data of the 3D scene or the resource data of the second intermediate state in the runtime environment provided by the runtime container is: The framework layer senses a second load event for the 3D scene's resource data or a second edit event for the application logic code, executes a function entity of at least one first type of API corresponding to the second load event or the second edit event, and when a second target API, which is a second type of API encapsulated in the function entity, is executed, provides the second target API to the runtime environment layer. The runtime environment layer includes the step of calling the corresponding software engine in the software library layer based on the second target API to respond to the second load event or second edit event, The method according to 9, characterized in that the first type of API is obtained by encapsulating the second type of API, and the second type of API is an API corresponding to a software engine included in the software library layer.

11. An electronic device comprising memory and a processor, wherein a computer program is stored in the memory, and the processor is coupled with the memory and executes the computer program to realize the steps of the method according to any one of claims 6 to 10.

12. A computer-readable storage medium in which a computer program is stored, characterized in that when the computer program is executed by a processor, the processor becomes capable of realizing the steps of the method according to any one of claims 6 to 10.

Images