XR application editing method, device, and storage medium

By introducing runtime containers into the XR application development system, providing a runtime environment, the complexity problem of game engine embedded 3D scene development in 2D applications is solved, and XR application development and operation across platforms and devices is realized, simplifying The development process and performance are improved.

WO2024244663A1PCT designated stage expired Publication Date: 2024-12-05TAOBAO CHINA SOFTWARE
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/084512
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-26
Filing Date
2024-03-28
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

In the prior art, game engines for terminal R&D engineers are difficult to adapt to the development of XR applications, and game engines have problems such as waste of resources, difficulty in UI interaction, and difficulty in integrating with SDKs with 2D applications when embedded in 3D scenes in 2D applications. This leads to complex and inconvenient development of XR applications.

Benefits of technology

It lowers the threshold for XR application development, simplifies the development process, solves cross-device operation problems, supports XR application development with embedded 3D scenes in independent forms and 2D applications, and improves development efficiency and performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024084512_05122024_PF_FP_ABST
    Figure CN2024084512_05122024_PF_FP_ABST
Patent Text Reader

Abstract

Embodiments of the present application provide an XR application editing method, a device, and a storage medium. In the embodiments of the present application, a runtime container is integrated in a content generator, and application content of an XR application is generated on the basis of a runtime environment provided by the runtime container; when application content is generated, the XR application is generated according to the application content and the runtime container, and the runtime container runs between the application content and the operating system, to provide a runtime environment for the application content. The runtime container is added between the application content and the operating system, so that there is no longer a strong dependency between running of the application content and the operating system, and development and operation of XR applications can be cross-platform and cross-device, making the development of XR applications simple and easy to implement, and solving the problems of XR application development and cross-device operation.
Need to check novelty before this filing date? Find Prior Art

Description

XR application editing method, device and storage medium

[0001] Cross-references

[0002] This application refers to Chinese patent application No. 2023106207762, filed on May 26, 2023, entitled “XR application editing method, device and storage medium”, which is incorporated into this application in its entirety by reference. Technical Field

[0003] The present application relates to the field of Internet technology, and in particular to an XR application editing method, device, and storage medium. Background Art

[0004] Extended Reality (XR) is a collective term for Augmented Reality (AR) and Virtual Reality (VR). VR uses computers to generate a virtual world that directly imparts visual, auditory, and tactile sensations to participants, allowing them to interactively observe and manipulate the virtual world. AR, on the other hand, cleverly integrates virtual information with the real world. Using multimedia, 3D modeling, intelligent interaction, and sensing, it simulates computer-generated virtual information such as text, images, 3D models, audio, and video, and then applies it to the real world, thereby enhancing the real world.

[0005] With the widespread adoption of XR technology, the supply of XR applications based on XR technology has become a pressing issue. Existing technologies include game development engines for three-dimensional (3D) game developers and two-dimensional (2D) application development engines for terminal R&D engineers. However, these development engines are not suitable for XR application development. Therefore, a development solution that can develop XR applications is urgently needed.

[0006] Summary of the Invention

[0007] Various aspects of the present application provide an XR application editing method, device, and storage medium to solve the development problems of XR applications.

[0008] An embodiment of the present application provides an XR application development system for terminal development, including: generating a runtime container required by an XR application, for providing a runtime environment during the generation and operation of the XR application;

[0009] A content generator, which integrates the runtime container and is used to generate application content of the XR application based on the runtime environment provided by the runtime container;

[0010] An application generation platform, configured to generate the XR application based on the application content and the runtime container;

[0011] During the running of the XR application, the runtime container runs between the application content and the operating system of the target device where the XR application is located, and the application content runs in the runtime environment provided by the runtime container.

[0012] An embodiment of the present application provides an XR application development method for terminal development, including: pre-developing and generating a runtime container required for the XR application, the runtime container being used to provide a runtime environment during the generation and operation of the XR application; in a content generator integrated with the runtime container, generating application content of the XR application based on the runtime environment provided by the runtime container; generating the XR application based on the application content and the runtime container; wherein, during the operation of the XR application, the runtime container runs between the application content and the operating system of the target device where the XR application is located, and the application content runs in the runtime environment provided by the runtime container.

[0013] The present application also provides a content generator that integrates a runtime container required for generating an XR application. The runtime container is used to provide a runtime environment during the generation or operation of the XR application. The content generator further includes:

[0014] A 3D scene effect editing module is configured to respond to an XR application generation operation, load 3D scene static resources, and edit 3D scene dynamic information for the 3D scene static resources based on a runtime environment provided by the runtime container to obtain 3D scene resource data;

[0015] a 3D application logic editing module, configured to edit application logic code for the 3D scene resource data based on a runtime environment provided by the runtime container, so as to obtain application content of the XR application;

[0016] The application content and the runtime container form the XR application, and the runtime container in the XR application provides a runtime environment for the application content during the running of the XR application.

[0017] An embodiment of the present application also provides an XR application editing method, which is applied to a content generator, wherein a runtime container is integrated in the content generator, and the runtime container is used to provide a runtime environment during the generation process or operation of the XR application. The method includes: loading 3D scene static resources in response to an XR application generation operation; adding 3D scene dynamic information to the 3D scene static resources based on the runtime environment provided by the runtime container to obtain 3D scene resource data; editing application logic code for the 3D scene resource data based on the runtime environment provided by the runtime container to obtain application content of the XR application; wherein the application content and the runtime container form the XR application, and the runtime container in the XR application provides a runtime environment for the application content during the operation of the XR application.

[0018] The embodiment of the present application further provides a runtime container for providing a runtime environment during the generation and operation of an XR application, including: a framework layer, a runtime environment layer, and a software library layer;

[0019] The software library layer includes multiple software engines and provides second-class application programming interfaces (APIs) of the multiple software engines, so as to encapsulate the first-class APIs according to the second-class APIs;

[0020] The framework layer is configured to provide application developers with the first-category API, so that the application developers can write application logic code for the XR application based on the first-category API, and to sense target events during the generation or operation of the XR application, run at least one function entity of the first-category API corresponding to the target event, and provide the target API encapsulated in the function entity to the runtime environment layer, where the target API belongs to the second-category API;

[0021] The runtime environment layer is used to call the corresponding software engine in the software library layer according to the target API to respond to the target event.

[0022] An embodiment of the present application also provides an XR application running method, including: in response to a trigger operation to run an XR application, starting a runtime container in the XR application to provide a runtime environment for application content in the XR application, the runtime container running between the application content and the operating system of the target device where the XR application is located; running the application content based on the runtime environment provided by the runtime container to run the XR application.

[0023] An embodiment of the present application also provides an XR content running method, which is applied to a host application embedded with an XR runtime container. The method includes: during the operation of the host application, displaying access entry information of the XR content; responding to a triggering operation on the access entry information, starting the XR runtime container and obtaining the XR content; and running the XR content based on the runtime environment provided by the XR runtime container.

[0024] An embodiment of the present application provides an electronic device, including: a memory and a processor; the memory is used to store a computer program, and the processor is coupled to the memory to execute the computer program to implement the steps in the XR application editing method, XR application running method, XR application development method, or XR content running method provided in the embodiment of the present application.

[0025] An embodiment of the present application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor is enabled to implement the steps of the XR application editing method, XR application running method, XR application development method, or XR content running method provided in the embodiment of the present application.

[0026] In an embodiment of the present application, a runtime container is integrated into a content generator, and application content for an XR application is generated based on the runtime environment provided by the runtime container. When generating application content, an XR application including the application content and the runtime container is generated based on the application content and the runtime container. During the operation of an XR application developed based on the runtime container, the runtime container runs between the application content and the operating system, and the runtime container provides a runtime environment for the application content, enabling the application content to run in the runtime environment. Since the runtime container is added between the application content and the operating system, the operation of the application content is no longer strongly dependent on the operating system, making the development and operation of XR applications cross-platform and cross-device. This makes the development of XR applications simple and easy to implement, solving the problems of developing and running XR applications across devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0028] FIG1 is a schematic diagram of the structure of an XR application development system for terminal development provided in an embodiment of the present application;

[0029] FIG2 is an implementation architecture of a runtime container provided by this embodiment;

[0030] FIG3 is a schematic diagram showing the relationship between the three-layer architecture of the runtime container provided in an embodiment of the present application;

[0031] FIG4 is a schematic diagram of a framework of an XR application runtime provided by an embodiment of the present application;

[0032] FIG5 is a schematic diagram of a 3D scene embedded in a host 2D application according to an embodiment of the present application;

[0033] FIG6 a is a schematic diagram of the initialization process of an XR application provided in an embodiment of the present application;

[0034] FIG6 b is a schematic diagram of a process of user-triggered function call according to an embodiment of the present application;

[0035] FIG6 c is a schematic diagram of a process of user-triggered event callback according to an embodiment of the present application;

[0036] FIG6 d is a schematic structural diagram of a content generator provided in an embodiment of the present application;

[0037] FIG7 a is a flowchart of an XR application development method provided in an embodiment of the present application;

[0038] FIG7 b is a schematic diagram of a flow chart of an XR application editing method provided in an embodiment of the present application;

[0039] FIG8 is a schematic diagram of a flow chart of a method for running an XR application according to an embodiment of the present application;

[0040] FIG9 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0042] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0043] With the popularization of XR technology, existing technologies are facing the problem of developing XR applications. Especially before the large-scale popularization of XR devices, the form of XR applications is mainly 2D applications with 3D scenes embedded. This makes the development of XR applications mainly for terminal R&D engineers for a long time. Terminal R&D engineers refer to engineers who develop 2D applications for mobile terminals such as mobile phones, including but not limited to: client engineers and front-end engineers.

[0044] To address XR application development challenges, some application developers, including terminal R&D engineers, have attempted to develop XR applications using game engines. A game engine is a framework for rapid game development. Common game engines include, but are not limited to, Unity and UE4. Designed specifically for game development, game engines are powerful and cover various functional modules, such as rendering, physics, and animation. Complete and complex XR applications can be developed based on these engines. However, developing XR applications based on game engines also has the following drawbacks:

[0045] First, terminal R&D engineers face enormous challenges in transitioning to game development. These challenges include at least the following:

[0046] (1) Game engines have high requirements for terminal R&D engineers, who need to have a deep understanding of 3D graphics knowledge, which is a relatively high threshold.

[0047] (2) You need to be familiar with the game engine. Game engines such as UE / Unity involve hundreds of header files and thousands of application programming interfaces (APIs), which are quite difficult to familiarize yourself with.

[0048] (3) The integrated development environment (IDE) of the game engine, such as UE blueprint, Unity editor and other tool chains, is designed independently and is quite different from the IDE for application development. It is difficult for terminal R&D engineers to adapt to it in a short period of time.

[0049] Second, the game engine is not designed for embedded apps, which makes it difficult to support the development of applications that embed 3D scenes in 2D applications. The following problems may exist:

[0050] (1) Multiple instances: The most prominent feature of 2D applications is the ability to switch between different pages. Pages are isolated from each other, support state cache recovery, and support back, so each page is an independent instance. However, game engines are exclusive and lack lifecycle management. Resources are difficult to recycle after page switching, resulting in resource waste. Memory resources can easily reach the upper limit of the app's usability, triggering application crashes.

[0051] (2) User Interface (UI) interaction: The UI of 2D applications is responsive rendering, while the UI of game engines is immediate rendering. The implementation mechanisms of the two are different, which makes it difficult for applications developed by game engines to have UI layout and smoothness, and it is also difficult to achieve complex UI effects.

[0052] (3) Problems with interaction with the software development kit (SDK) of 2D applications: Due to the complexity of the game engine architecture, it is impossible to integrate the SDK of 2D applications during the development of 3D scenes. Therefore, data communication between 3D scenes developed based on the game engine and 2D applications can only be achieved through events, which makes code implementation more complicated.

[0053] In view of the above, an embodiment of the present application provides an XR application development system for terminal development, which includes a runtime container that can provide a runtime environment during the XR application generation process (the XR application generation process can also be called the XR application development process) and the operation process, a content generator responsible for content generation, and an application generation platform responsible for packaging applications; the runtime container is integrated in the content generator, and the application content of the XR application is generated based on the runtime environment provided by the runtime container; when generating application content, the application generation platform generates an XR application containing the application content and the runtime container based on the application content and the runtime container; during the operation of the XR application developed based on the runtime container, the runtime container runs between the application content and the operating system, and the runtime container provides a runtime environment for the application content, so that the application content can run in the runtime environment. Since the runtime container is added between the application content and the operating system, the underlying information of the device can be shielded, so that the operation of the application content and the operating system are no longer strongly dependent, so that the development and operation of the XR application can be cross-platform and cross-device, thereby making the development of the XR application simple and easy to implement, solving the problem of development and cross-device operation of the XR application.

[0054] Furthermore, developing XR applications based on runtime containers enables the development and operation of XR applications across platforms and devices. This allows terminal R&D engineers to develop both independent XR applications for XR devices and XR applications that mainly embed 3D scenes in 2D applications. This can meet the needs of XR devices for independent XR applications and the needs of terminal devices for XR applications that mainly embed 3D scenes in 2D applications. It solves the pain points faced by the XR field across devices (such as mobile phones, AR devices, and VR devices) and adapts to the development of XR technology.

[0055] Furthermore, the XR application development system for terminal development provided in the embodiment of the present application pre-develops a runtime container that can provide a runtime environment, and provides a content generator and an application generation platform. For developers, they only need to develop application content based on the runtime container, and then use the application generation platform to package the application content and the runtime container into an XR application. It has the advantages of low threshold, parallel research and development, and complete research and development support. It has low requirements for developers and is suitable for both terminal research and development engineers and other developers such as game engineers. In addition, due to the existence of the runtime container, the development of XR applications is more pure, and various application requirements can be directly considered without being subject to various restrictions of the game engine. It can overcome the various defects and problems faced by the development of XR applications based on game engines mentioned above, and then develop XR applications with cross-device and cross-platform characteristics.

[0056] Furthermore, since the runtime environment is provided by the runtime container, the development of application content is relatively independent. The application content mainly includes 3D scene resource data and application logic code. During specific implementation, the application code is written in dynamic languages ​​such as JS. The XR applications developed in this way also have the characteristics of fast iteration and high performance.

[0057] In the various embodiments of this application, the implementation form of XR devices is not limited; they can be either AR devices or VR devices. AR devices include, but are not limited to, AR glasses and AR helmets; VR devices include, but are not limited to, VR glasses and VR helmets. Accordingly, terminal devices running 2D applications with embedded 3D scenes can include, but are not limited to, mobile phones, tablets, laptops, and smart bracelets.

[0058] The following embodiments of this application will describe in detail the XR application development system for terminal development provided by the embodiments of this application in conjunction with the accompanying drawings.

[0059] FIG1 is a schematic diagram of the structure of an XR application development system 100 for terminal development provided in an embodiment of the present application. As shown in FIG1 , the system 100 includes: a content generator 20, a runtime container 30, and an application generation platform 40.

[0060] As shown in Figure 1, the XR application development system 100 of this embodiment includes a runtime container 30. The runtime container 30 is a pre-developed software stack that runs on an operating system. It does not belong to the operating system. The software stack is provided for XR application development and is required for the development of XR applications. Its role is to provide a runtime environment during the development and operation of XR applications. The runtime environment refers to the environment required for the operation of the developed XR application or the intermediate program code in the XR application development process, such as some software engines and environment variables required for the operation of the XR application or the intermediate program code, as well as the operation logic required for the operation of the XR application or the intermediate program code. In this embodiment, the software engine is a broad concept, which is a summary of various engines, library functions, etc.

[0061] On the basis that the XR application development system 100 includes a runtime container 30, the XR application development process of this embodiment includes: a process of developing application content required for the XR application based on the runtime environment provided by the runtime container 30 and a process of generating the XR application based on the runtime container and the application content. Among them, the content generator 20 is responsible for generating the application content of the XR application based on the runtime environment provided by the runtime container 30. In order to facilitate the content generator 20 to use the runtime environment provided by the runtime container 30, the runtime container 30 is integrated in the content generator 20. In this embodiment, the application content is developed based on the runtime environment provided by the runtime container 30, so that the application content has the ability to run in the runtime environment provided by the runtime container 30, and can achieve consistency between the application content effect during development and the application content effect at runtime.

[0062] When generating application content, the application generation platform 40 is responsible for generating an XR application based on the runtime container 30 and the application content. In this embodiment, the XR application is different from a traditional application program. It includes both application content and a runtime container 30 that is responsible for providing a runtime environment for the application content. Since the application content is developed based on the runtime environment provided by the runtime container 30, the application content has the ability to run in the runtime environment provided by the runtime container 30. Based on this, the runtime container 30 and the application content are integrated together to form an XR application. In this way, during the operation of the XR application, the runtime container 30 can run between the application content and the operating system of the target device where the XR application is located, and is responsible for providing a runtime environment for the application content, so that the application content runs in the runtime environment provided by the runtime container, and ultimately achieves the purpose of normal operation of the XR application. Among them, the target device where the XR application is located can be an XR device or a terminal device running a host 2D application, and there is no limitation on this.

[0063] In this embodiment, the runtime container 30 runs on the operating system of the target device where the XR application is located, and uses the operating system to provide a runtime environment for the application content of the XR application. Specifically, it means using the various software and hardware resources of the target device with the help of the operating system of the target device, such as CPU, GPU, memory, network card, bandwidth and other resources, and providing the application content of the XR application with the runtime environment required for its operation based on these software and hardware resources, so that the application content runs in the runtime environment, and then presents 3D scenes and interaction and / or control based on 3D scenes to the user.

[0064] In this embodiment, the implementation form of the XR application can be an independent application form or an application form that mainly embeds a 3D scene in a 2D application. Depending on the implementation form of the XR application, the application generation platform 40 generates the XR application in different ways. The role of the application generation platform 40 is to package the runtime container 30 into a corresponding installation package based on the implementation form of the XR application and publish it so that the target device can locally install the runtime container 30 based on the corresponding installation package.

[0065] If the XR application is a standalone application, the application generation platform 40 can integrate the runtime container 30 and the application content into the XR application installation package and publish the XR application installation package. This allows the XR application to be downloaded, installed, and run like other standalone applications. Specifically, the XR application can be an AR application running on an AR device or a VR application running on a VR device, without limitation.

[0066] If the XR application is primarily a 2D application with a 3D scene embedded within it, the 2D application that requires the embedded 3D scene is referred to as the host 2D application. The application generation platform 40 can integrate the runtime container 30 into the host 2D application's installation package and publish the host 2D application's installation package. This allows the runtime container 30 to be downloaded, installed, and run like any other standalone application, and the runtime container 30 to be downloaded, installed, and run as well. The XR application's application content can be published independently of the host 2D application, or embedded within the host 2D application's installation package and published along with it. This is not a limitation. Optionally, considering the large amount of data in the application content of the XR application, the application content of the XR application can be published separately to the resource server corresponding to the host 2D application. In this way, during the operation of the host 2D application, when it is necessary to run the 3D scene provided by the XR application, the application content of the XR application can be downloaded from the resource server on demand, and the application content can be run in the runtime environment provided by the runtime container 30, thereby presenting the 3D scene in the page of the host 2D application, wherein the 3D scene is embedded in a page of the host 2D application.

[0067] It should be noted that the embodiments of the present application are not limited to host 2D applications, and can be any 2D application with 3D scene display requirements, such as e-commerce applications, entertainment applications, teaching applications, or medical applications.

[0068] The runtime container 30 of this embodiment is a provider of a runtime environment running on the operating system. Whether in the XR application development process or in the XR application operation process, the runtime container 30 provides the runtime environment in the same or similar manner. As shown in Figure 2, an implementation architecture of the runtime container 30 provided in this embodiment includes: a framework layer 31, a runtime environment layer 32, and a software library layer 33. The following describes the three layers separately:

[0069] Libraries layer 31: Contains a series of basic libraries required for XR applications. These basic libraries provide various software engines and / or library functions, such as but not limited to: 3D rendering engine, AI reasoning engine, engine or library function for implementing download function provided by download library, engine or library function for implementing storage function provided by storage library, engine or library function for implementing network function provided by network library, engine or library function for implementing configuration function provided by configuration library, blockchain engine, engine or library function for implementing tracking function provided by tracking library, 2D UI engine and event engine, etc. Further, depending on the downloaded content, the download library can be divided into picture library, video library, etc. These software engines are responsible for providing XR applications with some basic capabilities required for their operation, such as 3D scene rendering capability, reasoning capability of neural network models, download capability of various resources, storage capability, network communication capability, configuration capability, etc. The following description of various engines and software libraries is only an example and is not limited to this:

[0070] 3D rendering engine: responsible for providing 3D rendering, animation, physical simulation and other capabilities, such as but not limited to: AceNNR.

[0071] AI inference engine: Responsible for providing end-side inference and execution capabilities for deep neural network models, such as MNN.

[0072] Blockchain engine: responsible for providing blockchain services.

[0073] Network library: Provides efficient network transmission capabilities between the end side (mainly the target device where the XR application is located) and the cloud side (mainly the cloud server corresponding to the XR application).

[0074] Download library: Based on the network library and combined with the cloud-side encapsulated download library, it provides the ability to download various resources from the cloud side and cache resources;

[0075] Image library: Based on the network library and combined with the cloud-side encapsulated image download library, it supports image downloads in multiple compression formats and provides a local image caching mechanism.

[0076] Video library: Based on the network library and combined with the video download library encapsulated by the cloud-side server, it supports video downloads in multiple formats and provides a local video caching mechanism.

[0077] Tracking database: Responsible for providing log tracking services, compressing the log data generated during the operation of XR applications on the client side and uploading it to the cloud side;

[0078] Repository: Provides file storage services. If the repository is a relational database, it provides relational storage services. If the database is a time series database, it provides time series storage services, etc.

[0079] Configuration library: responsible for providing various configuration functions for 3D models, coordinates, events, etc. in XR applications.

[0080] In this embodiment, the basic capabilities provided by the various software engines mentioned above can be encapsulated into APIs, and these basic capabilities are exposed to the outside through these APIs. These APIs are called basic APIs. Relative to the first type of API, these basic APIs can also be called second type APIs. The first type of API is a type of API exposed to application developers for application developers to write application logic code. The second type of API does not need to be exposed to application developers. The first type of API is encapsulated based on the second type of API. Each function entity corresponding to the first type of API may contain one or more second type APIs, and there is no limitation on this. Application developers develop application logic code based on the first type of API. During the operation of the XR application, when these application logic codes are run, they first trigger the call of the first type of API, and then trigger the call of the second type of API encapsulated in the first type of API, and then trigger the call of the corresponding software engine, and then use the basic capabilities provided by the software engine to achieve 3D scene presentation, etc.

[0081] The Framework layer 32 provides the capabilities required for XR application development, primarily exposing Class 1 APIs to application developers. The implementation of Class 1 APIs is largely provided by the software engine in the Libraries layer. The number and types of Class 1 APIs provided by the Framework layer vary depending on the application scenario. The following examples include, but are not limited to, the following:

[0082] (1) 3D scene API: This type of API is used for 3D scene development. Moreover, this type of API mainly encapsulates the 3D rendering capabilities of the 3D rendering engine in the Libraries layer, so that application developers do not have to deal with the 3D rendering engine. To a certain extent, it can lower the threshold for application developers to understand and develop XR applications.

[0083] There are multiple 3D scene APIs. Here are some examples of 3D scene APIs:

[0084] XRScene: represents a 3D scene. This API can be used to implement a 3D scene.

[0085] XRModel: represents a 3D model. This API can be used to implement a 3D model in a 3D scene.

[0086] XRAnchor: represents a reference coordinate in a 3D world. This API can be used to achieve position calibration in a 3D scene.

[0087] XREvent: represents system and application-related events. Through this API, you can respond to various events in the 3D scene and respond accordingly.

[0088] XRConfig: Represents the configuration of XR applications. This API allows you to configure various XR applications, such as configuring different resource loading and operation strategies for XR applications for high-end, medium-end, and low-end devices. You can also configure the resolution, layer, and type of each 3D model or other object in the 3D scene.

[0089] (2) Lifecycle Management API: Responsible for managing the lifecycle of XR applications, including but not limited to launching, exiting, and updating XR applications. This type of API is mainly implemented by encapsulating the lifecycle management capabilities of the lifecycle management engine in the Libraries layer.

[0090] (3) Resource Management API: This API is responsible for managing various resources of XR applications, including but not limited to downloading, caching, and storage of resources required by XR applications. This type of API mainly encapsulates the download and storage capabilities provided by the download engine and storage engine in the Libraries layer.

[0091] (4) UI Interaction API: This type of API is applied to the application form of embedding 3D scenes in 2D applications. It is responsible for managing the UI interaction between XR applications and their host 2D applications, and supports the display of pages developed in the W3C way on the front end. This UI interaction API can be implemented as a web container, which is responsible for handling the display function of the visual UI of the host 2D application in the XR application. It is a physical container responsible for providing the GUI of the 2D application. Alternatively, it can be implemented as a Native container, which is a virtualized container. This type of API mainly encapsulates the UI interaction capabilities provided by the 2D UI engine in the Libraries layer.

[0092] (5) Event Interaction API: This API manages event interactions within XR applications, including but not limited to click, long press, zoom, rotation, and proximity. This API primarily encapsulates the event management and response capabilities provided by the event engine in the Libraries layer.

[0093] In addition to exposing the first-class API to application developers, the Framework layer 32 also has some logical functions. For example, during the generation or operation of the XR application, it is responsible for sensing the target event. The target event refers to an event involving the call of at least one first-class API. According to the operation logic between at least one first-class API, the function entity of at least one first-class API is run, and the target API encapsulated in the function entity is provided to the runtime environment layer 33. The target API here is the second-class API encapsulated in the function entity of the first-class API. For different first-class APIs, the functions they implement are generally different, so the second-class APIs encapsulated in their function entities will also be different, and there is no limitation on this.

[0094] Runtime environment layer 33: Responsible for providing a runtime virtual machine (VM) adapted to the programming language used by XR applications. These runtime VMs are responsible for providing a runtime environment for XR applications. Specifically, they are used to call the corresponding software engine in the Libraries layer to respond to the target events perceived by the Framework layer 32 according to the target API provided by the Framework layer 32, so as to complete the operation of intermediate resource data in the XR application development process or complete the operation of application content in the XR application.

[0095] Among them, the VM adapted to the programming language used by the XR application can be a JS VM or a Python VM. If the application logic code of the XR application is written in JS, a JS VM is provided; if the application logic code of the XR application is written in Python, a Python VM is provided. It should be noted that these runtime VMs provide a runtime environment for XR applications. On the one hand, they use the target device's operating system to use the target device's hardware and software resources to provide the XR application with the required hardware and software resources. On the other hand, based on these hardware and software resources, they call the basic capabilities provided by the corresponding software engine in the Libraries layer to render 3D scene resource data and interpret and execute the application logic code, allowing XR applications to use the various basic capabilities in the Libraries layer and ultimately present 3D scenes and 3D scene-based interaction and / or control processes to users.

[0096] JS VM: This primarily supports application developers (such as terminal R&D engineers) in writing JS application logic code, dynamically distributing it and interpreting and executing it through the JS VM to dynamically update the logic of XR applications. For example, 3D scene UI interfaces and interaction logic.

[0097] Python VM: This primarily supports application developers (such as terminal R&D engineers) writing Python application logic code, dynamically distributing it and interpreting and executing it through the Python VM. This allows for dynamic updates of Python-written application code (such as specific algorithmic tasks). For example, in AR scenarios, Python can be used to write application logic code, specifically, facial recognition algorithms for AR applications.

[0098] It should be noted that in the embodiments of the present application, the number of runtime VMs included in the runtime environment layer 33 is not limited, and the number of runtime VMs can be one or more. For example, a JS VM or a Python VM can exist alone, or both a JS VM and a Python VM can exist. When there are multiple runtime VMs, different runtime VMs are implemented using different dynamic programming languages ​​to provide runtime environments for XR applications implemented using different dynamic programming languages.

[0099] Furthermore, when there are multiple runtime VMs, the runtime environment layer 33 also includes a VM scheduling module for scheduling, during the execution of the XR application, a target VM implemented in the same dynamic programming language as the XR application among the multiple runtime VMs, so that the target VM provides a runtime environment for the XR application. Accordingly, the VM scheduling module is further configured to schedule, during the development of the XR application, a target VM implemented in the same dynamic programming language as the XR application among the multiple runtime VMs, so that the target VM provides a runtime environment for intermediate resource data during the development of the XR application.

[0100] In this embodiment, there is an association relationship between the three-layer architecture of the runtime container 30, which is shown in Figure 3. The Libraries layer 31 is located at the bottom layer, and the runtime environment layer 33 is located between the Libraries layer 31 and the Framework layer 32. The Libraries layer 31 is responsible for providing various basic capabilities. The basic capabilities provided by different software engines are different. For example, the 3D rendering engine provides 3D rendering capabilities, and the AI ​​reasoning engine provides AI model reasoning capabilities based on deep neural networks. These basic capabilities are encapsulated to form different second-class APIs for capability exposure. On the one hand, the second-class API is exposed to the runtime environment layer 33. On the other hand, the second-class API is further encapsulated as a first-class API and transmitted to application developers through the Framework layer 32. The Framework layer 32 can detect target events that trigger first-category API calls during XR application development or operation. For example, these events can trigger XR application execution during XR application development, loading 3D scene static resources, viewing 3D scene effects after adding 3D scene dynamic information, or viewing interactive effects after adding application code. These target events trigger the call of at least one first-category API and the execution of at least one first-category API function entity. When a second-category API contained in the function entity is executed, the second-category API is provided to the runtime environment layer 33. The runtime environment layer 33 can receive the second-category API provided by the Framework layer 32 during XR application development or operation, and use the second-category API to call the basic capabilities provided by the software engine in the Libraries layer 31. For details about loading 3D scene static resources, viewing 3D scene effects after adding 3D scene dynamic information, and viewing interactive effects after adding application code, please refer to the following description.

[0101] In this embodiment, various basic capabilities in the Libraries layer can be implemented in C++, which is a cross-platform language. By implementing various basic capabilities in the Libraries layer based on C++, XR applications developed based on the runtime environment provided by the runtime container have cross-platform and cross-device features.

[0102] In this embodiment, when various basic capabilities in the Libraries layer are implemented in C++, the following methods may be used to expose the basic capabilities:

[0103] The first method is the native exposure method, which requires the application logic code in the XR application to be developed in C++. This exposure method does not have the ability to quickly iterate.

[0104] The second method is non-native transparency, which means that the application logic code in the XR application can be written in JS or Python. With the advantages of JS or Python, the application logic can be dynamically updated and iterated.

[0105] Furthermore, in the case of adopting a non-native transparency method, as further shown in FIG3 , the following two methods can be used to implement API transparency:

[0106] (1) In the Libraries layer, the second-class API is exposed to the JS VM or Python VM in the runtime environment layer through JS Binding or Python Binding technology. The JS VM or Python VM encapsulates the second-class API into the first-class API, and then exposes the first-class API to the Framework layer;

[0107] (2) In the Libraries layer, the second-class API is exposed to the JS VM or Python VM in the runtime environment layer 33 through JS binding or Python binding technology, and the second-class API is encapsulated into the first-class API, and the first-class API is directly exposed from the Libraries layer to the Framework layer.

[0108] In the embodiment of the present application, the application content of the XR application mainly includes 3D scene resource data and application logic code. Among them, 3D scene resource data refers to the resource data responsible for presenting the 3D scene, for example, including: 3D scene static resources and 3D scene dynamic information. 3D scene static resources refer to static resources in the 3D scene, such as various 3D models, images, etc.; 3D scene dynamic information refers to some dynamic information related to the 3D scene effect, such as the movement trajectory of the 3D model, the camera system in the 3D scene, the water system in the 3D scene, etc. These dynamic information can make the 3D scene effect richer and more diverse. The application logic code data includes various code data that reflect the application logic. These application logics include logic for controlling and interacting with various resources in the 3D scene from the application level, such as movement control of character models, interactive control with character models or object models, and the appearance of some special effects when specific conditions are met.

[0109] In an optional embodiment, 3D scene static resources are pre-generated. Based on these resources, 3D scene dynamic information and application logic code are edited on the 3D scene static resources to obtain the application content of the XR application. Based on this, one way for the content generator 20 to generate application content includes: responding to an XR application generation operation, loading the 3D scene static resources, and adding the 3D scene dynamic information and application logic code to the 3D static scene resources based on the runtime environment provided by the runtime container 30 to obtain the application content of the XR application.

[0110] Among them, the application logic code can be written by the application developer based on the first type of API provided to the application developer by the Framework layer in the runtime container 30.

[0111] In an optional embodiment, as shown in FIG1 , the XR application development system 100 provided in this embodiment further includes a 3D model development tool 10 for model developers, enabling them to use the 3D model development tool 10 to develop 3D scene static resources upon which XR applications are generated. Specifically, the 3D model development tool 10 can generate various 3D models in response to 3D model development operations by the model developers, and output these 3D models as at least part of the 3D scene static resources.

[0112] 3D scene static resources include various static resources required to build 3D scenes in XR applications, including but not limited to:

[0113] (1) 3D model file, used to describe the 3D model. The 3D model can be classified according to application requirements, and can be classified differently in different application scenarios. Taking the XR scene shown in FIG5 as an example, the 3D model can be divided into, but not limited to, three categories: characters, objects, and scenes. In other words, the characters, shoes, bags, etc. shown in FIG5 are all 3D models, and the entire 3D scene is also a 3D model. In this embodiment, the format of the 3D model is not limited, and can be, for example, but not limited to: FBX, OBJ, GItf, etc.

[0114] (2) Image resources refer to images that need to be presented at certain locations in the 3D scene.

[0115] (3) Video resources refer to video information that needs to be presented at certain locations in the 3D scene.

[0116] (4) Audio resources refer to audio information that needs to be played at certain locations in the 3D scene.

[0117] It should be noted that the above-mentioned 3D model files are required resources, while image resources, video resources, and audio resources are optional resources, which can be determined according to the application scenario.

[0118] In this embodiment, the implementation method of the 3D model development tool 10 is not limited. Any development tool that can provide code editing, code compilation, debugging and graphical interactive interface for 3D models is applicable to the embodiment of this application. For example, Android Studio or XCode can be used to write plug-ins, or a self-developed IDE for XR application development can be used. The self-developed IDE needs to have the following functions: code editing, code compilation, debugging and graphical interactive interface for 3D models, etc.

[0119] After obtaining the 3D scene static resources required for developing XR applications, the content generator 20 can be used to add 3D scene dynamic information and application logic code to these static resources to ultimately obtain the application content of the XR application. The application content of the XR application can also be called the resource package of the XR application.

[0120] In this embodiment, as shown in Figure 1 , one input to the content generator 20 is a 3D scene static resource. Based on the runtime environment provided by the runtime container 30 , 3D scene dynamic information is added to the 3D scene static resource to obtain 3D scene resource data containing the 3D scene static resource and the 3D scene dynamic information. Furthermore, as shown in Figure 1 , another input to the content generator 20 is application logic code. Based on the runtime environment provided by the runtime container 30 , the application logic code is added to the 3D scene resource data to obtain the application content of the XR application.

[0121] In this embodiment, the internal implementation architecture of content generator 20 is not limited. As shown in Figure 6d, one implementation structure of content generator 20 includes: a 3D scene effect editing module 21, a 3D application logic editing module 22, a content output module 23, and a runtime container 30. In terms of the hierarchical architecture, 3D scene effect editing module 21 and 3D application logic editing module 22 are located above runtime container 30, requiring runtime container 30 to provide a runtime environment for them.

[0122] The 3D scene effect editing module 21 is used to respond to the XR application generation operation, load the 3D scene static resources, and edit the 3D scene dynamic information for the 3D scene static resources based on the runtime environment provided by the runtime container 30 to obtain 3D scene resource data.

[0123] The 3D application logic editing module 22 is in communication with the 3D scene effect editing module 21 and is used to load 3D scene resource data and edit application logic code for the 3D scene resource data based on the runtime environment provided by the runtime container 30 to obtain application content of the XR application.

[0124] The content output module 23 is communicatively connected to the 3D application logic editing module 22, and is used to output the application content of the XR application to the application generation platform 40, so that the application generation platform 40 generates the XR application based on the application content and the runtime container 30. The application content of the XR application includes 3D scene resource data and application logic code. Among them, the 3D scene resource data includes but is not limited to: the organization description file of the 3D scene, mesh data, material data, texture data, video data and other information. Among them, the organization description file of the 3D scene includes but is not limited to: 3D scene organization, lighting settings, rendering settings, camera movement, physical collisions, etc. Material data includes but is not limited to: glass material, metal material and other information. Among them, the application logic code data can be written in JS language but is not limited to, and can be expressed as xxx.js.

[0125] During the application content generation process, the role of the runtime container 30 is to provide a runtime environment for the intermediate data that appears in the XR application generation process, so that the intermediate data can present the running effect, and then determine whether it is necessary to adjust the 3D scene dynamic information or application logic code based on the running effect, and finally obtain application content with running effects that meet the requirements.

[0126] For the 3D scene effect editing module 21, it is specifically used to: respond to the loading operation of the 3D scene static resource, start the runtime container 30, and run the 3D scene static resource in the runtime environment provided by the runtime container 30 to obtain a 3D static picture; respond to the editing operation on the 3D static picture, edit the 3D scene dynamic information for the 3D scene static resource to obtain the first intermediate state resource data; respond to the scene effect viewing operation, re-call the runtime container 30, run the first intermediate state resource data in the runtime environment provided by the runtime container 30, and obtain the first intermediate state 3D picture; when the first intermediate state 3D picture does not meet the scene effect, adjust the 3D scene dynamic information to meet the scene effect to obtain the 3D scene resource data.

[0127] Optionally, the 3D scene effect editing module 21 is used to respond to the project creation operation, create the project, and name the project. For example, the project name can be 3D scene, such as AAA; respond to the import operation of 3D scene static resources, import 3D scene static resources, these static resources include but are not limited to: various 3D models, etc.; start the runtime container 30, run the 3D scene static resources in the runtime environment provided by the runtime container 30, and display a 3D static picture on the interactive interface, which contains these 3D models. In this embodiment, the content generator provides an interactive interface, which is a 3D interface with three coordinate axes xyz. Preferably, the 3D scene itself is a 3D model. When displaying the 3D scene, the center of the 3D scene can be located at the origin of the xz plane of the editing interface, and the y value of the ground plane is 0. Further, on this basis, the 3D scene effect editing module 21 edits dynamic information for these 3D models, and ultimately obtains target 3D scene resource data that meets the requirements.

[0128] In this embodiment, the content generator 20 provides an interactive interface and can also display a first intermediate 3D image. Furthermore, in response to editing operations on the 3D static image, 3D scene dynamic information is edited for the 3D scene static resource, including: displaying an editing control on the 3D static image; in response to triggering operations on the editing control, displaying a dynamic information editing interface, wherein various editable information items are displayed on the dynamic information editing interface; in response to configuration operations on these information items, obtaining the configured information of these information items as 3D scene dynamic information, and superimposing this 3D scene dynamic information with the 3D scene static resource to form 3D scene resource data.

[0129] Depending on the XR application scenario, the interface form of the above-mentioned dynamic information editing interface and the information items displayed will be different, and there is no limitation on this. To illustrate the information items: for example, it can include information items for configuring trigger actions, and the triggering methods that can be configured for this information item include but are not limited to: sliding trigger, long press trigger, drag trigger and multiple click trigger; for another example, it includes information items for configuring 3D model display effects, and the display methods that can be configured for this information item include but are not limited to: animation, sound effects, joining a meeting, etc.; for another example, it includes information items for configuring the ending form of the 3D model, and the ending forms that can be configured for this information item include but are not limited to: hiding at the end, closing at the end, looping, etc. In addition, according to the application scenario, you can also edit relevant dynamic information such as camera control, lighting control, water control, scene placeholders, and pathfinding systems for 3D scene static resources.

[0130] The following is an example of the process of editing the aforementioned 3D scene dynamic information for 3D scene static resources:

[0131] (1) Camera Control: In response to camera control editing operations for a 3D model, relevant camera control information is edited for the 3D model, so as to provide the 3D model with rich camera control capabilities in the 3D scene. The camera control information edited for the 3D model includes, but is not limited to, control information for controlling the 3D model to switch between different perspectives, control information for controlling the 3D model to move on a preset camera track, and control information for controlling the 3D model to follow other 3D models or other scene objects. In this way, when presenting a 3D scene, when these camera control information are encountered, the 3D rendering engine can be called to execute these camera control information, controlling the 3D model to switch between different perspectives, move on a preset track, or follow other 3D models or other scene objects, etc.

[0132] (2) Lighting control: The 3D rendering engine provided by the runtime container 30 supports both real-time and baked lighting. Real-time lighting supports directional light sources, while baked lighting supports directional light, point light, spot light, and area light. In response to a lighting control editing operation on a 3D model, the runtime container 30 edits relevant lighting control information for the 3D model, such as selecting a light source type for the 3D model and editing parameters such as the light source's color, illumination angle, illumination range, and illumination intensity, so as to render the required lighting effects for the 3D model in the 3D scene.

[0133] (3) Water Control: The 3D rendering engine provided by the runtime container 30 supports water materials. Using water materials based on plane meshes can quickly achieve water surface rendering effects. Under the Material option, select Shader Graphs or Water Shader to use the water material. In this embodiment, the parameters involved in the water material include but are not limited to: Deep Color: deep water color; Shallow Color: shallow water color; Strength: currently inactive; Smoothness: smoothness; Displacement: wave movement range; Normal Steength: wave normal strength; Normal Tiling: wave normal map tiling coefficient. In response to the water control editing operation for the 3D model, the relevant water control information is edited for the 3D model, such as selecting a water shader for the 3D model, editing parameters such as water color, smoothness, wave movement range, and wave normal strength, so as to render the 3D model with a water surface rendering effect that meets the requirements in the 3D scene.

[0134] (4) Scene placeholder editing: Placeholder information needs to be added to the 3D scene, such as placeholder information for 3D models, placeholder information for cameras, and placeholder information for pictures, videos, or other objects for use by the application layer. The specific location in the 3D scene where the placeholder information should be set varies depending on the application's Product Requirements Document (PRD). For example: 1) Image placeholder is used to mark the location where the image needs to be placed in the 3D scene; 2) Video placeholder is used to mark the location where the video needs to be placed in the 3D scene; 3) Red envelope placeholder is used to mark the location where the red envelope needs to be displayed in the 3D scene; 4) Camera placeholder, multiple camera placeholders can be defined in the 3D scene, so that the camera perspective can be switched on these camera placeholders at runtime; 5) 3D model placeholder, used to mark the position of the corresponding 3D model in the 3D scene, which can realize operations such as model movement, rotation, and scaling, and can also support subsequent updates of the 3D model; 6) Clickable area placeholder, the 3D model in the 3D scene can mark the clickable area outside the 3D model in the 3D scene and the location for collision detection in the 3D scene by adding a clickable component. In response to the placeholder editing operation for the 3D scene, relevant placeholder information is edited for the 3D scene. For example, placeholder information such as 3D model placeholders, picture placeholders, video placeholders, red envelope placeholders, camera placeholders, and clickable area placeholders are set in the 3D scene, so as to display 3D models, pictures, videos, red envelopes and other objects in reasonable positions in the 3D scene and perform rendering effects such as switching the camera perspective of the 3D model in the camera placeholder.

[0135] (5) Pathfinding system editing: Three different areas are set according to the map of the 3D scene: 1) Walkable: Indicates that the current area allows specific model objects (such as character models) to walk on it, such as the ground; 2) Not Walkable: Indicates obstacles (such as walls), indicating that specific model objects cannot pass through; 3) Jump: Indicates that the current area allows specific model objects to jump over, such as thresholds or ravines. In response to the pathfinding editing operation for the 3D scene, the relevant pathfinding information is edited for the 3D scene, for example, setting areas in the 3D scene that allow specific model objects to pass through, areas that do not allow them to pass through, or areas that can be skipped, so as to present the pathfinding rendering effect of the specific model objects in the 3D scene.

[0136] It is explained here that in the above-mentioned process of editing the 3D scene static resources, the 3D scene effect editing module 21 in the content generator 20 can start the runtime container 30, and run the first intermediate resource data obtained after adding the above-mentioned dynamic information in the runtime environment provided by the runtime container 30. For example, the 3D rendering engine provided by the runtime container 30 can be called to render the first intermediate resource data to obtain a first intermediate 3D picture. The above-mentioned dynamic information is continuously corrected according to the rendering effect of the first intermediate 3D picture until a rendering effect that meets the requirements is obtained. The dynamic information at this time is used as the final dynamic information. The final dynamic information and the 3D scene static resources form the 3D scene resource data in the application content.

[0137] In an optional embodiment, the above-mentioned dynamic information editing interface can be independent of the interface for presenting the 3D static image, or the above-mentioned dynamic information editing interface can be implemented embedded in the interface for presenting the 3D static image, and there is no limitation on this. Regardless of the interface form, a refresh control or an effect viewing control can be set on the above-mentioned dynamic information editing interface or the interface for presenting the 3D static image. The application developer can initiate the operation of viewing the 3D scene effect after adding the 3D scene dynamic information by clicking or long pressing these controls. The 3D scene effect editing module 21 can respond to the scene effect viewing operation and re-call the runtime container 30, run the first intermediate state resource data in the runtime environment provided by the runtime container 30, and obtain the first intermediate state 3D image. The first intermediate state 3D image reflects the 3D scene effect after adding the 3D scene dynamic information to the 3D scene static resource.

[0138] In this embodiment, the application developer can manually determine whether the first intermediate state 3D picture satisfies the scene effect. If it is determined that the first intermediate state 3D picture does not meet the scene effect, the application developer can adjust the 3D scene dynamic information, regenerate the first intermediate state resource data, and run the regenerated first intermediate state resource data in the runtime environment provided by the runtime container 30 until the first intermediate state 3D picture that meets the scene effect is obtained. At this time, the 3D scene dynamic information and the 3D scene static resources form the 3D scene resource data. Alternatively, a functional module or component for scene effect recognition and judgment of various 3D pictures can be generated, and the functional module or component automatically captures the picture state of the first intermediate state 3D picture and determines whether the first intermediate state 3D picture meets the scene effect based on the picture state.

[0139] For the 3D application logic editing module 22, it is specifically used to: respond to the loading operation of 3D scene resource data, start the runtime container 30, and run the 3D scene resource data in the runtime environment provided by the runtime container 30 to obtain a first intermediate 3D picture; respond to the editing operation of the application logic code, generate second intermediate resource data according to the 3D scene resource data and the application logic code, and re-call the runtime container 30, and run the second intermediate resource data in the runtime environment provided by the runtime container 30 to obtain a second intermediate 3D picture; when the second intermediate 3D picture does not meet the interactive effect, adjust the application logic code to meet the interactive effect to obtain the application content of the XR application.

[0140] In this embodiment, the running effect of the second intermediate state resource data, that is, the running effect of the application logic code in the 3D scene is divided into two categories, one is scene control and the other is scene interaction.

[0141] Scene control: Controlling the 3D model in the 3D scene based on user input. This means that the application logic code has the function of responding to user input and controlling the 3D model in the 3D scene based on the user input. The method of controlling the 3D model in the 3D scene is not limited; for example, the 3D model can be controlled to move, jump, or switch camera perspectives. For example, in the scene shown in Figure 5, the movement of the character model can be controlled based on the user's operation of the joystick (an example of user input). The new position coordinates of the character model are calculated based on the user input, such as the world coordinates [0.15, 0.22, 0.31], and the character model is moved from the original position coordinates to the new position coordinates. It should be noted that the user input method will vary depending on the target device displaying the 3D scene. If the target device is a mobile terminal such as a mobile phone, the user can input through gestures, voice, or touch. If the target device is a VR device or AR device, the user can input through buttons on the controller or input through posture. Taking touch input as an example, it can be operations such as a user clicking, long pressing, or double-clicking on the 3D scene interface, or the user triggering operations on certain controls in the 3D scene.

[0142] Scene interaction: This primarily refers to triggering various response events or interactive information based on user input or user-controlled 3D models. This means that the application logic code has the ability to respond to user input or monitor the state of the 3D model being manipulated, triggering response events or interactive information based on user input or the state of the 3D model. For example: 1) A user clicks on a 3D model to control the display or hiding of the 3D model, or to display related special effects; 2) When a user manipulates a 3D model close to a certain area or object, a red envelope message or other interactive information is displayed in the 3D scene.

[0143] Based on the above, when the second intermediate state 3D picture does not meet the interactive effect, exemplary cases of adjusting the application logic code include but are not limited to: when the position of the 3D model is not ideal, modifying the position of the interaction with the 3D model in the application logic code, the trigger operation type of the 3D model, etc.; when displaying the red envelope information in the 3D scene does not meet the application requirements, modifying the red envelope information in the application logic code to flip card information, etc., until the application logic code that meets the interactive effect is obtained. At this time, the application logic code and the 3D scene resource data form the application content of the XR application.

[0144] In this embodiment, the content generator 20 provides an interactive interface on which the first intermediate 3D image and the second intermediate 3D image can be displayed. Similarly, regarding the method for determining whether the second intermediate 3D image satisfies the interactive effect, reference can be made to the method for determining whether the first intermediate 3D image satisfies the scene effect mentioned above, and will not be repeated here.

[0145] In this embodiment, the application logic code is pre-written by the application developer. In addition to providing the runtime environment, the runtime container of this embodiment can also provide the application developer with the API required for developing XR applications. For the sake of ease of description and distinction, these APIs for application developers are referred to as the first-class APIs. Application developers can write the application logic code required for XR applications based on the first-class APIs. Among them, the first-class API is obtained by encapsulating the second-class API. The second-class API is the API provided to the outside by the software engine provided by the runtime container 30. The software engine provided by the runtime container can be called through the second-class API, and then these software engines provide intermediate resource data or the runtime environment required for the XR application. Since the first-class API is obtained by encapsulating the second-class API, its number is much smaller than that of the second-class API, and the first-class API is more concrete and easy to understand, so that application developers can write application logic code more simply and efficiently, reducing the difficulty of developing XR applications.

[0146] It should be noted that the manner in which the aforementioned 3D scene static resources, 3D scene resource data, first intermediate resource data, and second intermediate resource data are executed in the runtime environment provided by runtime container 30 may vary depending on the implementation architecture of runtime container 30. This will be described in detail below with reference to the implementation architecture of runtime container 30 shown in FIG2 .

[0147] When editing 3D scene dynamic information:

[0148] The 3D scene effect editing module 21 responds to the loading operation of the 3D scene static resources and starts the runtime container 30. After the runtime container 30 is started, the Framework layer perceives the loading event of the 3D scene static resources, which is a specific example of the aforementioned target event, and determines at least one first-class API corresponding to the event, such as but not limited to: 3D scene API, lifecycle management API, and resource management API; then, based on the logical relationship between these APIs, the function entities of these APIs are executed, and the function entities of these APIs encapsulate the second-class API corresponding to the 3D rendering engine, for example, the function entity of the 3D scene API encapsulates the second-class API corresponding to the 3D rendering engine, the function entity of the lifecycle management API encapsulates the second-class API corresponding to the lifecycle management engine, and the function entity of the resource management API encapsulates the second-class API corresponding to the resource management engine. The second-class API encapsulated in these function entities is called the target API. When any target API is executed, the Framework layer provides the target API to the runtime environment layer, and the runtime environment layer calls the corresponding software engine according to the target API to respond to the event of loading the 3D scene static resources. Exemplarily, the runtime environment layer calls the 3D rendering engine to render the 3D scene static resources to obtain a 3D static image, and calls the lifecycle management engine to manage the lifecycle of the 3D static image, such as displaying and closing; and calls the resource management engine to perform resource management on the 3D scene static resources, such as local storage, caching, etc.

[0149] The 3D scene effect editing module 21 responds to the scene effect viewing trigger operation and re-calls the runtime container 30. After the runtime container 30 is re-called, the Framework layer perceives the event of viewing the 3D scene effect after adding 3D scene dynamic information. This event is a specific example of the aforementioned target event and determines at least one first-class API corresponding to the event, such as but not limited to: 3D scene API, lifecycle management API, and resource management API. Then, based on the logical relationship between these APIs, the function entities of these APIs are executed. The function entities of these APIs encapsulate the second-class API corresponding to the 3D rendering engine, for example, the function entity of the 3D scene API encapsulates the second-class API corresponding to the 3D rendering engine, the function entity of the lifecycle management API encapsulates the second-class API corresponding to the lifecycle management engine, and the function entity of the resource management API encapsulates the second-class API corresponding to the resource management engine. The second-class API encapsulated in these function entities is called the target API. When any target API is executed, the Framework layer provides the target API to the runtime environment layer, and the runtime environment layer calls the corresponding software engine according to the target API to respond to the event of loading 3D scene static resources. Exemplarily, the runtime environment layer calls the 3D rendering engine to render the first intermediate state resource data to obtain the first intermediate state 3D picture, and calls the lifecycle management engine to manage the lifecycle of the first intermediate state 3D picture, such as display and closing; and calls the resource management engine to perform resource management on the first intermediate state resource data, such as local storage, caching, etc.

[0150] While editing the application logic code:

[0151] The 3D application logic editing module 22 responds to the loading operation of the 3D scene resource data and starts the runtime container 30. After the runtime container 30 is started, the Framework layer perceives the loading event of the 3D scene resource data, which is a specific example of the aforementioned target event, and determines at least one first-class API corresponding to the event, such as but not limited to: 3D scene API, lifecycle management API, resource management API, UI interaction API, and event interaction API; then, based on the logical relationship between these APIs, the function entities of these APIs are executed, and the function entities of these APIs encapsulate the second-class APIs, such as the function entity of the 3D scene API encapsulates the second-class API corresponding to the 3D rendering engine, the function entity of the lifecycle management API encapsulates the second-class API corresponding to the lifecycle management engine, the function entity of the resource management API encapsulates the second-class API corresponding to the resource management engine, the function entity of the UI interaction API encapsulates the second-class API corresponding to the 2D UI engine, and the function entity of the event interaction API encapsulates the second-class API corresponding to the event management engine. The second type of API encapsulated in these function entities is called the target API. When any target API is executed, the Framework layer provides the target API to the runtime environment layer, and the runtime environment layer calls the corresponding software engine according to the target API to respond to the loading event of the 3D scene static resource. For example, the runtime environment layer calls the 3D rendering engine to render the 3D scene resource data to obtain the first intermediate state 3D picture, and calls the life cycle management engine to manage the life cycle of the first intermediate state 3D picture, such as displaying and closing; and calls the resource management engine to perform resource management on the 3D scene resource data, such as local storage, caching, etc., and further the event engine responds to interactive events during the operation of the 3D scene resource data, such as responding to interactive events such as long press, click, rotation, etc. on the interface. When the XR application is implemented as a 3D scene embedded in the host 2D application, the application logic code may contain the function of UI interaction with the host 2D application, and the 2D UI engine can respond to the UI interaction between the application logic code and the host 2D application and respond to it.

[0152] The 3D application logic editing module responds to the editing operation of the application logic code and re-calls the runtime container 30. After the runtime container 30 is re-called, the Framework layer perceives the editing event of the application logic code, which is a specific example of the aforementioned target event, and determines at least one first-class API corresponding to the event, such as but not limited to: 3D scene API, lifecycle management API, resource management API, UI interaction API, and event interaction API, etc.; then, according to the logical relationship between these APIs, the function entities of these APIs are executed, and the second-class API is encapsulated in the function entities of these APIs. The second-class API encapsulated in these function entities is called the target API. When any target API is executed, the Framework layer provides the target API to the runtime environment layer, and the runtime environment layer calls the corresponding software engine according to the target API to respond to the event of loading 3D scene static resources. For detailed description, please refer to the above embodiment and will not be repeated here.

[0153] Based on the above operations, the application content of the XR application can be generated, and then the application generation platform 40 packages the application content with the runtime container to generate the XR application. According to the application form of the XR application, different methods can be used to deploy it to the target device. After the XR application is deployed to the target device, the XR application can be run on the target device. Regardless of the method, the running process of the XR application refers to starting the runtime container 30 in the XR application. The runtime container 30 runs between the application content of the XR application and the operating system of the target device where the XR application is located. The runtime container 30 provides a runtime environment for the application content of the XR application, runs the application content in the runtime environment, and then presents the 3D scene and the interaction or control in the 3D scene to the user. Among them, the framework of the XR application runtime is shown in Figure 4. The top layer is the application content of the XR application, the middle is the runtime container 30, and the bottom layer is the operating system of the target device that carries the XR application. The runtime container 30 runs on the operating system of the target device and provides a runtime environment for the application content of the XR application. The application content of the XR application runs in this runtime environment, presents the 3D scene to the user and interacts and / or controls based on the 3D scene. The interaction here includes user-triggered interaction, as well as interaction and / or control automatically triggered within the 3D scene according to application requirements.

[0154] In the above framework, the Framework layer 32 of the runtime container 30 perceives the target event during the running of the XR application, runs the function entity of at least one first-class API corresponding to the target event, and provides the target API encapsulated in the function entity to the runtime environment layer 33. The target API refers to the second-class API encapsulated in the function entity. The runtime environment layer 33 is used to call the corresponding software engine in the Libraries layer 31 to respond to the target event based on the target API provided by the Framework layer 32. Specifically, the runtime environment layer 33 can call the corresponding software engine in the Libraries layer 31 to respond to the target event based on the target API, and transparently transmit the response result to the application interface of the XR application through the Framework layer 32.

[0155] During the execution of an XR application, the target events that the Framework layer 32 may perceive include, but are not limited to, trigger events for launching the XR application, interaction events with target objects within the XR application, and trigger events for closing the XR application. Target objects within an XR application include, but are not limited to, at least one of a 3D model, a 2D object, and an interactive area within a 3D scene. As shown in Figure 5 , a 3D model can be a character model, an object model, or an entire 3D scene model; correspondingly, 2D objects include, but are not limited to, images, audio, or video that need to be displayed within the 3D scene.

[0156] In an optional embodiment, the XR application is implemented as an independent application form, and the user can click, double-click, or long press the icon of the XR application to initiate a trigger operation to run the XR application; the Framework layer 32 can respond to the trigger operation of the XR application icon and generate a trigger event to run the XR application.

[0157] In another optional embodiment, the XR application is implemented as an application form in which a 3D scene is embedded in a host 2D application. In this application form, an access portal to the 3D scene is provided in the host 2D application, and the user can click, double-click, or long press the access portal to initiate a trigger operation to run the XR application; the Framework layer 32 can respond to the trigger operation of the access portal of the XR application and generate a trigger event to run the XR application.

[0158] Depending on the application scenario to which the XR application belongs, the first category API corresponding to the trigger event of running the XR application will be different, and the embodiments of the present application do not limit this. For example: the first category API corresponding to the trigger event of running the XR application includes: 3D scene API, lifecycle management API and resource management API, and the function entities of these APIs encapsulate the second category API corresponding to the software engine. For example, the function entity of the 3D scene API encapsulates the second category API corresponding to the 3D rendering engine, the function entity of the lifecycle management API encapsulates the second category API corresponding to the lifecycle management engine, and the function entity of the resource management API encapsulates the second category API corresponding to the resource management engine. The Framework layer 32 runs the function entities of the first category API in sequence according to the application logic, and when running to the second category API encapsulated in these function entities, it uses the second category API as the target API and provides it to the runtime environment layer 33. The runtime environment layer 33 responds to the trigger event of running the XR application by calling the corresponding software engine according to the target API. Exemplarily, the runtime environment layer 33 calls the 3D rendering engine to call the lifecycle management engine to download the resource package of the XR application, calls the 3D rendering engine to render the downloaded resources, and feeds back the download and resource loading status to the Framework layer 32 through the runtime environment layer to complete the 3D scene initialization of the XR application. During this process, the resource management engine can also be called to manage the resource package of the XR application, such as local storage, caching, etc.

[0159] Depending on the application scenario to which the XR application belongs, the first category API corresponding to the interaction event of the target object in the XR application will be different, and the embodiments of the present application do not limit this. For example: at least one first category API corresponding to the interaction event, for example, includes but is not limited to: 3D scene API, UI interaction API and event interaction API, etc., and the function entity of these APIs encapsulates the second category API corresponding to the 3D rendering engine, for example, the function entity of the 3D scene API encapsulates the second category API corresponding to the 3D rendering engine, the function entity of the UI interaction API encapsulates the second category API corresponding to the 2D UI engine, and the function entity of the event interaction API encapsulates the second category API corresponding to the event management engine. The Framework layer 32 runs the function entities of the first category API in sequence according to the application logic, and when running to the second category API encapsulated in these function entities, it uses the second category API as the target API and provides it to the runtime environment layer 33. The runtime environment layer 33 responds to the interaction event by calling the corresponding software engine according to the target API. Exemplarily, the runtime environment layer calls the 3D rendering engine to render the resource data involved in the interactive event, and calls the event engine to respond to the interactive event, such as responding to interactive events such as long press, click, rotation, etc. on the interface, and when the interactive event includes UI interaction with the host 2D application, it can also call the 2D UI engine to respond to the UI interaction with the host 2D application.

[0160] For the runtime environment layer 33, the method of calling the corresponding software engine in the libraries layer 31 to respond to the target event will vary depending on the target API. The following examples illustrate this with some software engines included in the libraries layer 31:

[0161] When the target API includes the second type of API corresponding to the download engine in the Libraries layer 31, calling the download engine to download the application content of the XR application from the server;

[0162] When the target API includes the second type of API corresponding to the storage engine in the Libraries layer 31, calling the storage engine to perform storage management on the application content of the XR application;

[0163] When the target API includes the second type of API corresponding to the lifecycle management engine in the Libraries layer 31, calling the lifecycle management engine to perform lifecycle management on the XR application;

[0164] When the target API includes the second type of API corresponding to the 3D rendering engine in the Libraries layer 31, calling the 3D rendering engine to perform 3D rendering on the XR application;

[0165] When the target API includes the second type of API corresponding to the AI ​​inference engine in the Libraries layer 31, calling the AI ​​inference engine to perform AI inference processing on the XR application;

[0166] When the target API includes the second type of API corresponding to the 2D UI engine in the Libraries layer 31, the 2D UI engine is called to process the UI interaction between the XR application and the host 2D application;

[0167] When the target API includes the second type of API corresponding to the event engine in the Libraries layer 31, the event engine is called to process the event callback logic in the XR application.

[0168] In the following embodiments, an XR application is an application in which a 3D scene is embedded in a host 2D application. Assuming the host 2D application is an e-commerce app, to increase the fun and user experience of the e-commerce app, a 3D scene can be embedded in the e-commerce app, such as an AR fitting scene or a VR shopping scene. Using this scenario as an example, the operation process of the runtime container provided in the embodiments of this application is described.

[0169] An access portal for the XR application is provided on a certain page of the e-commerce app. After the user clicks on the access portal, they enter the 3D scene space presented by the XR application, as shown in Figure 5. In this embodiment, the application content (or resource package) of the XR application includes 3D scene resource data and application logic code written in JS. It is developed based on the above-mentioned development system. During the operation of the e-commerce app, when the user triggers access to the 3D scene space of the XR application, the three-layer architecture in the runtime container can be called. This three-layer architecture supports the display and interaction of the 3D scene space, including but not limited to the following aspects:

[0170] 1. XR application lifecycle management

[0171] The runtime container responds to the trigger operation of the XR application, calls the API corresponding to the download library in the second category of APIs, downloads the resource package of the XR application, completes the resource loading of the XR application, and calls the API corresponding to the 3D rendering engine in the second category of APIs, uses the 3D rendering engine to render the 3D scene for the XR application, so as to display the 3D scene space on the relevant pages of the e-commerce APP, and responds to the exit or close operation of the XR application, and performs actions such as cleaning up related resources (such as memory, CPU) when the XR application exits, so as to realize the management of the XR application life cycle.

[0172] Furthermore, the management process of the XR application lifecycle is explained in conjunction with the three-layer architecture of the runtime container. As shown in Figure 6a, the process of initializing an XR application includes: the framework layer responds to the trigger operation of entering the 3D scene space, enters the 3D scene, triggers the runtime VM to initialize the 3D scene, and the runtime VM calls the download library in the software library layer to download the XR application's resource package; the runtime VM calls the 3D rendering engine to render the 3D scene based on the downloaded resources, updates the resource download and rendering status, and provides the resource download and rendering status to the framework layer; after the XR application's resource package download is complete, the download library notifies the runtime VM of the resource package download completion; the runtime VM notifies the framework layer of the completion of 3D scene initialization; and then the framework layer completes UI operations on the 3D scene. It should be noted that the XR application's resource package can be encrypted and compressed. Therefore, before rendering, the runtime VM can also call the corresponding engine to decrypt and decompress the resource package.

[0173] 2. The user triggers a first-category operation in the 3D scene space. The first-category operation refers to an operation that triggers a function call. The called function is responsible for responding to the first-category operation and making a response to the first-category operation.

[0174] Taking the 3D scene space shown in Figure 5 as an example, the first type of operations triggered by the user include but are not limited to the following:

[0175] The user operates the joystick in the 3D scene to move the character model. This operation triggers a series of behaviors, such as position and scene changes in the roaming state, which can be responded to and processed by callback functions;

[0176] The user manipulates the camera's perspective, which triggers scene changes that can be responded to and processed by callback functions.

[0177] The user clicks the screen to trigger the play function, which will trigger the playing of music or special effects and map switching, etc. These changes can be responded to and processed by the callback function.

[0178] Furthermore, the processing of user-triggered function calls is explained in conjunction with the three-tier architecture of the runtime container. As shown in Figure 6b, the processing of the first type of user-triggered operation includes:

[0179] Case 1: In response to the user's operation of moving the character model, the framework layer notifies the runtime VM of the user's operation of moving the character model. The runtime VM calls the 3D rendering engine in the software library layer to re-render the position of the character model.

[0180] Case 2: In response to the user's operation of switching the camera perspective, the framework layer notifies the runtime VM of the user's operation of switching the camera perspective. The runtime VM calls the 3D rendering engine in the software library layer to re-render the 3D scene or 3D model to obtain the 3D scene or 3D model at the new perspective.

[0181] Case 3: The framework layer responds to the user's operation of playing music and special effects, and notifies the runtime VM of the user's operation of playing music and special effects. The runtime VM calls the multimedia library and 3D special effects library in the software library layer to play music and special effects.

[0182] 3. The user triggers the second type of operation in the 3D scene space. The second type of operation refers to the operation that triggers the event callback. The called back event is responsible for responding to the second type of operation and responding to the second type of operation.

[0183] Taking Figure 5 as an example, situations in which a user initiates the second type of operation to trigger an event callback include but are not limited to: the user operates the character model to move, the 3D rendering engine performs collision detection, and when it finds that the character model approaches or leaves the specified model object, it triggers an event callback to the upper application logic code layer for response, such as popping up a red envelope, coupon or venue message.

[0184] Furthermore, the processing of user-triggered event callbacks is explained in conjunction with the three-tier architecture of the runtime container. As shown in Figure 6c, the processing of the second type of user-triggered operation includes:

[0185] Case 1: The VM calls the 3D rendering engine in the software library layer to monitor whether the 3D model (taking the character model as an example) collides with the set object, which is referred to as physical collision detection; when it detects that the character model is close to the preset object, the runtime VM feeds back to the framework layer; the framework layer executes the corresponding code data in the target application logic code data in response, such as displaying red envelope information.

[0186] Case 2: The VM calls the 3D rendering engine in the software library layer to monitor whether the 3D model (taking the character model as an example) collides with the set object, which is referred to as physical collision detection; when it detects that the character model leaves the preset object, it feeds back to the framework layer through the runtime VM; the framework layer calls the corresponding code data in the target application logic code data to respond, such as canceling the display of red envelope information.

[0187] The XR application development system provided in the embodiments of this application has the following beneficial effects:

[0188] (1) Cross-device and high performance: Implement the basic capabilities in the Libraries layer through C++ and perform targeted optimization to achieve better performance, such as parallel computing acceleration of the device's instruction set, to achieve cross-device operation from the code execution level; further, when encapsulating these basic capabilities to obtain APIs, secondary encapsulation can be performed on the APIs of different devices, so that XR applications can also achieve cross-device operation at the interaction level, thereby improving the performance of XR applications;

[0189] (2) Fast iteration: Choose to use dynamic programming languages ​​such as Python or JS to implement the application logic code and runtime VM of XR applications, and use the runtime VM as the glue layer to call the basic capabilities in the underlying Libraries layer implemented in C++. The iteration frequency of the basic capabilities in the Libraries layer is low and can be iterated through version updates. However, the update frequency of the application logic of XR applications is high. It is implemented in a dynamic programming language. With the advantages of dynamic programming languages, it is conducive to the rapid release and update of application logic while ensuring global high performance.

[0190] (3) Lightweight: Based on the development requirements of XR applications, the software engines or software libraries in the Libraries layer are lightweighted. For example, some unnecessary software libraries are discarded, and some software libraries are introduced through grafting rather than direct transplantation. In addition, the software engines or software libraries in the Libraries layer are released dynamically instead of being released along with the installation package of the XR application or the host 2D application, further achieving lightweightness. Furthermore, when an AI inference engine is included, an MNN engine is used to implement image processing, matrix calculation and other operations based on MNN, which helps reduce the package size of the end-side AI inference engine and further achieve lightweightness.

[0191] (4) Low threshold: By encapsulating the basic capabilities in the Libraries layer and exposing the first-class API to application developers through the Framework layer, it abstractly encapsulates the basic capabilities such as rendering, physics, and animation provided by the 3D rendering engine, and provides a few concepts such as XRScene, XRModel, XRAnchor, XREvent, and XRConfig, which are easy for XR application developers to understand and use.

[0192] (5) As the XR application runs, the stability of the XR application can be monitored, and multi-dimensional data such as the crash information, startup performance, and page performance of the XR application can be obtained in real time. Based on these monitoring data, the XR application can be operated and maintained in the later stage. These operation and maintenance operations include but are not limited to: patch repair, information configuration, etc.

[0193] In this embodiment, a runtime container is used to construct an editing tool chain suitable for embedding 3D scenes in a host 2D application, namely, a content generator 20. The structure and function of the content generator 20 can be found in the embodiment shown in FIG6d above and will not be repeated here. Specifically, the content generator 20, based on the runtime container, can meet the constraints of package size, performance, and effects required for embedding in a 2D application, and has the following effects:

[0194] (1) Efficiency is improved, greatly reducing the cost of art and technical communication and collaboration. By integrating the runtime container into the content generator, the 3D rendering engine in the runtime container's software library layer can achieve what you see is what you get editing of 3D scene effects, making it easier for application developers to edit 3D scene effects and application logic code.

[0195] (2) Consistent effects: The 3D content generator and the target device reuse the same runtime container, which can achieve consistency in content editing and target device operation effects.

[0196] (3) Standard constraints: The 3D content generator embeds custom 3D model standard specifications, involving geometry, materials, lighting, cameras, animation, etc. The final 3D scene export will be verified according to the 3D model standard, ensuring the standardization of the final precipitated 3D model and meeting the constraints such as the effect and performance of the XR application running on the target device.

[0197] In addition to providing an XR application development system, the embodiments of this application also provide an XR application development method, an XR application editing method, and an XR application running method. The following embodiments will describe the XR application development method, XR application editing method, and XR application running method respectively.

[0198] FIG7a is a flow chart of an XR application development method provided in an embodiment of the present application. The method is applied to the above-mentioned XR application development system, as shown in FIG7a , and includes:

[0199] 71a. Pre-develop the runtime container required to generate the XR application and integrate the runtime container into the content generator. The runtime container is used to provide a runtime environment during the XR application generation and operation process.

[0200] 72a. In a content generator integrated with a runtime container, generate application content of the XR application based on a runtime environment provided by the runtime container;

[0201] 73a. Generate an XR application based on the application content and the runtime container; wherein, during the running of the XR application, the runtime container runs between the application content and the operating system of the target device where the XR application is located, and the application content runs in the runtime environment provided by the runtime container.

[0202] In an optional embodiment, in an integrated runtime container content generator, based on the runtime environment provided by the runtime container, the application content of the XR application is generated, including: responding to an XR application generation operation, loading 3D scene static resources; based on the runtime environment provided by the runtime container, adding 3D scene dynamic information to the 3D scene static resources to obtain 3D scene resource data; based on the runtime environment provided by the runtime container, editing application logic code for the 3D scene resource data to obtain the application content of the XR application. For the detailed implementation of each step, please refer to the relevant description of the aforementioned system embodiment or the following method embodiment, which will not be repeated here.

[0203] In an optional embodiment, the XR application is generated based on the application content and the runtime container, including: integrating the runtime container and the application content into the installation package of the XR application and publishing the installation package of the XR application; and / or integrating the runtime container into the installation package of the host two-dimensional 2D application and publishing the installation package of the host 2D application, and publishing the application content to the resource server corresponding to the host 2D application, where the host 2D application is a 2D application that needs to embed the XR application.

[0204] Figure 7b is a flow chart of an XR application editing method provided in an embodiment of the present application. This method is applied to the content generator in the above-mentioned XR application development system. The content generator is integrated with a runtime container, which is used to provide a runtime environment during the XR application generation process or operation process. As shown in Figure 7b, the method includes:

[0205] 71b. Respond to 3D resource editing operations and load 3D scene static resources;

[0206] 72b. Based on the runtime environment provided by the runtime container, add 3D scene dynamic information to the 3D scene static resource to obtain 3D scene resource data;

[0207] 73b. Based on the runtime environment provided by the runtime container, edit the application logic code for the 3D scene resource data to obtain the application content of the XR application; the application content and the runtime container form an XR application, and the runtime container in the XR application provides a runtime environment for the application content during the running of the XR application.

[0208] In an optional embodiment, based on the runtime environment provided by the runtime container, 3D scene dynamic information is added to the 3D scene static resource to obtain 3D scene resource data, including:

[0209] In response to the loading operation of the 3D scene static resource, the runtime container is started, and the 3D scene static resource is run in the runtime environment provided by the runtime container to obtain a 3D static image;

[0210] In response to an editing operation on the 3D static image, editing 3D scene dynamic information for the 3D scene static resource to obtain first intermediate state resource data;

[0211] In response to the scene effect viewing operation, the runtime container is re-called, and the first intermediate state resource data is run in the runtime environment provided by the runtime container to obtain the first intermediate state 3D picture;

[0212] When the first intermediate 3D image does not satisfy the scene effect, the 3D scene dynamic information is adjusted to obtain the scene effect, so as to obtain 3D scene resource data.

[0213] In an optional embodiment, based on the runtime environment provided by the runtime container, application logic code is edited for the 3D scene resource data to obtain application content of the XR application, including:

[0214] In response to the loading operation of the 3D scene resource data, the runtime container is started, and the 3D scene resource data is run in the runtime environment provided by the runtime container to obtain a first intermediate 3D picture;

[0215] In response to the editing operation of the application logic code, second intermediate state resource data is generated according to the 3D scene resource data and the application logic code, and the runtime container is re-invoked to run the second intermediate state resource data in the runtime environment provided by the runtime container to obtain a second intermediate state 3D picture;

[0216] When the second intermediate 3D image does not satisfy the interactive effect, the application logic code is adjusted to satisfy the interactive effect to obtain the application content of the XR application.

[0217] In an optional implementation, the runtime container includes: a framework layer, a runtime environment layer, and a software library layer. For the architecture and working principle of the runtime container, please refer to the aforementioned embodiment and will not be repeated here.

[0218] Based on the above, running a 3D scene static resource in a runtime environment provided by a runtime container to obtain a 3D static image includes: a framework layer sensing a first loading event of the 3D scene static resource, and running at least one function entity of a first-category API corresponding to the first loading event; when running to a first target API encapsulated in the function entity, providing the first target API to a runtime environment layer, the first target API being a second-category API; and the runtime environment layer calling a corresponding software engine in a software library layer to respond to the first loading event based on the first target API;

[0219] Correspondingly, the first intermediate state resource data is run in the runtime environment provided by the runtime container to obtain the first intermediate state 3D picture, including: the framework layer perceives the first editing event on the 3D static picture, and runs the function entity of at least one first-class API corresponding to the first editing event; when running to the first target API encapsulated in the function entity, the first target API is provided to the runtime environment layer, and the first target API belongs to the second-class API; the runtime environment layer calls the corresponding software engine in the software library layer to respond to the first editing event according to the first target API.

[0220] Accordingly, running the 3D scene resource data in the runtime environment provided by the runtime container to obtain a first intermediate 3D picture includes:

[0221] The framework layer senses the second loading event of the 3D scene resource data, and runs the function entity of at least one first-class API corresponding to the second loading event; when running to the second target API encapsulated in the function entity, the second target API is provided to the runtime environment layer, and the second target API belongs to the second-class API; the runtime environment layer calls the corresponding software engine in the software library layer to respond to the second loading event based on the second target API.

[0222] Correspondingly, the second intermediate state resource data is run in the runtime environment provided by the runtime container to obtain a second intermediate state 3D picture, including: the framework layer perceives the second editing event of the application logic code, and runs the function entity of at least one first-class API corresponding to the second editing event; when running to the second target API encapsulated in the function entity, the second target API is provided to the runtime environment layer, and the second target API belongs to the second-class API; the runtime environment layer calls the corresponding software engine in the software library layer to respond to the second editing event according to the second target API.

[0223] FIG8 is a flow chart of a method for running an XR application provided in an embodiment of the present application. The method is applied to the runtime container in the above-mentioned XR application development system. As shown in FIG8 , the method includes:

[0224] 81. In response to a trigger operation to run the XR application, start a runtime container in the XR application to provide a runtime environment for application content in the XR application, where the runtime container runs between the application content and an operating system of a target device where the XR application is located;

[0225] 82. Run application content based on the runtime environment provided by the runtime container to run XR applications.

[0226] In an optional embodiment, before responding to the trigger operation of running the XR application, it also includes: installing the runtime container and application content of the XR application on the target device according to the installation package of the XR application, and the installation package of the XR application includes the runtime container and application content; accordingly, responding to the trigger operation of running the XR application can be responding to a trigger operation such as double-clicking or long pressing the icon of the XR application to trigger the running of the XR application.

[0227] and / or,

[0228] In an optional embodiment, before responding to the trigger operation of running the XR application, it also includes: installing a runtime container on the target device according to the installation package of the host 2D application, the host 2D application is a 2D application that needs to embed the XR application, and the installation package of the host 2D application includes a runtime container, and the application content is published to the resource server corresponding to the host 2D application for dynamic loading. During the operation of the host 2D application, the access entry information of the XR application can be displayed. Accordingly, the trigger operation in response to running the XR application can be a response to the trigger operation of the access entry information, such as clicking or long pressing. In addition, in this scenario, before running the application content based on the runtime environment provided by the runtime container, it also includes an operation of downloading the application content from the resource server.

[0229] In an optional embodiment, the runtime container includes: a framework layer, a runtime environment layer, and a software library layer. For the architecture and working principle of the runtime container, please refer to the aforementioned embodiment and will not be repeated here.

[0230] Accordingly, the application content is run based on the runtime environment provided by the runtime container to run the XR application, including:

[0231] The framework layer senses target events during the running of the XR application. The target event corresponds to at least one function entity of the first-category API. The framework layer provides the target API encapsulated in the function entity to the runtime environment layer. The target API belongs to the second-category API.

[0232] The runtime environment layer calls the corresponding software engine in the software library layer to respond to target events based on the target API provided by the framework layer to run the XR application;

[0233] The first type of API is obtained by encapsulating the second type of API, and the second type of API is the API corresponding to the software engine included in the software library layer.

[0234] The runtime environment layer calls the corresponding software engine in the software library layer to respond to the target event based on the target API provided by the framework layer, including at least one of the following operations:

[0235] When the target API includes the second type of API corresponding to the download engine in the software library layer, calling the download engine to download the application content of the XR application from the server;

[0236] When the target API includes a second-category API corresponding to the storage engine in the software library layer, calling the storage engine to perform storage management on the application content of the XR application;

[0237] When the target API includes a second-category API corresponding to the lifecycle management engine in the software library layer, calling the lifecycle management engine to perform lifecycle management on the XR application;

[0238] When the target API includes a second-category API corresponding to the 3D rendering engine in the software library layer, calling the 3D rendering engine to perform 3D rendering on the XR application;

[0239] When the target API includes the second-category API corresponding to the AI ​​inference engine in the software library layer, calling the AI ​​inference engine to perform AI inference processing on the XR application;

[0240] When the target API includes the second-category API corresponding to the 2D UI engine in the software library layer, the 2D UI engine is called to process the UI interaction between the XR application and the host 2D application;

[0241] When the target API includes the second type of API corresponding to the event engine in the software library layer, the event engine is called to process the event callback logic in the XR application.

[0242] In an optional embodiment, the runtime environment layer includes at least one runtime virtual machine (VM), where different runtime VMs are implemented using different dynamic programming languages ​​and are used to provide a runtime environment for XR applications implemented using different dynamic programming languages. Accordingly, when there are multiple at least one runtime VMs, the method further includes: the runtime environment layer scheduling a target VM among the multiple runtime VMs, which is implemented using the same dynamic programming language as the XR application, based on the dynamic programming language used by the XR application, so that the target VM serves the XR application.

[0243] In addition to the above embodiments, the embodiments of the present application also provide an XR content running method, which is applied to a host application with an embedded XR runtime container. The XR runtime container is located in the installation package of the host application and can be installed together with the installation of the host application. Among them, the implementation structure and function of the XR runtime container can refer to the runtime container in the aforementioned embodiment. In this embodiment, an XR runtime container is embedded in the host application to provide a runtime environment for the operation of XR content. By running the XR content in the runtime environment provided by the XR runtime container, the purpose of presenting the XR content in the host application is ultimately achieved. Based on this, the XR content running method includes: running the host application; during the operation of the host application, displaying the access entry information of the XR content, for example, the access entry information can be a link on an application page provided by the host application or an icon or control with a link function, etc., which is not limited to this; in response to the triggering operation of the access entry information, starting the XR runtime container and obtaining the XR content; running the XR content based on the runtime environment provided by the XR runtime container. The detailed implementation of running XR content based on the runtime environment provided by the XR runtime container is the same as or similar to the detailed implementation of "running the application content of the XR application based on the runtime environment provided by the runtime container" in the aforementioned embodiment, and will not be repeated here.

[0244] In the embodiments of the present application, the method for obtaining XR content is not limited. In an optional embodiment, the XR content is deployed on the resource server corresponding to the host application, and the method for obtaining the XR content includes: downloading the XR content from the resource server corresponding to the host application. In another optional embodiment, the XR content can also be embedded in the host application. For example, the XR content is located in the installation package of the host application and can be installed together with the installation of the host application. The method for obtaining the XR content includes: loading the XR content from the local cache.

[0245] In the embodiments of the present application, the XR content can be implemented as the application content of the XR application in the aforementioned embodiments; accordingly, the XR content is combined with the XR runtime container to be implemented as the XR application in the aforementioned embodiments.

[0246] The detailed implementation and beneficial effects of each step in the above method embodiment of the present application have been described in detail in the aforementioned embodiments and will not be elaborated here.

[0247] It should be noted that the execution entity of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution entity of steps 71 to 73 can be device A; for another example, the execution entity of steps 71 and 72 can be device A, and the execution entity of step 73 can be device B; and so on.

[0248] In addition, some of the processes described in the above embodiments and the accompanying drawings include multiple operations that appear in a specific order, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or may be executed in parallel. The sequence numbers of the operations, such as 71, 72, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to being different types.

[0249] FIG9 is a schematic structural diagram of an electronic device provided in an embodiment of the present application. As shown in FIG9 , the electronic device includes: a memory 1001 and a processor 1002 .

[0250] The memory 1001 is used to store computer programs and can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions, messages, pictures, videos, etc. for any application or method operating on the electronic device.

[0251] Processor 1002 is coupled to memory 1001 and is used to execute the computer program in memory 1001 to implement the steps in the XR application development method, XR application editing method, XR application operation method, or XR content operation method provided in the embodiments of the present application.

[0252] For a detailed description of the XR application development method, the XR application editing method, the XR application running method, or the XR content running method, please refer to the aforementioned embodiments and will not be repeated here.

[0253] Furthermore, as shown in Figure 9, the electronic device also includes other components such as a communication component 1003, a display 1004, a power supply component 1005, and an audio component 1006. Figure 9 only schematically shows some components, which does not mean that the electronic device only includes the components shown in Figure 9. When the electronic device of this embodiment executes the XR application editing method or the XR application running method, it can be implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone or an IOT device, or it can also be implemented as an AR device or a VR device. In the case where the electronic device is implemented as an AR device or a VR device, the electronic device may also include components such as an operating handle, which are not shown in Figure 9. When the electronic device of this embodiment executes the XR application development method, it can be implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, etc. that simultaneously runs a content generator and a runtime container, or it can be implemented as a traditional server, a cloud server, or a system in which the cloud and the terminal are matched with each other, without limitation.

[0254] Accordingly, an embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor is enabled to implement the steps in the above-mentioned XR application development method or XR application editing method or the steps in the XR application running method or XR content running method.

[0255] The above-mentioned memory can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random-access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0256] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0257] The above-mentioned display includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor can not only sense the boundary of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0258] The power supply assembly provides power to various components of the device in which the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply assembly is located.

[0259] The above-mentioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as call mode, recording mode, and voice recognition mode, the microphone is configured to receive external audio signals. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0260] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to magnetic disk storage, compact disc read-only memory (CD-ROM), optical storage, etc.) that contain computer-usable program code.

[0261] The present application is described with reference to the flow chart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present application. It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one flow chart flow or multiple flows and / or one box or multiple boxes of the block diagram.

[0262] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0263] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0264] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interfaces, network interfaces, and memory.

[0265] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0266] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media such as modulated data signals and carrier waves.

[0267] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0268] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A content generator, It is characterized in that A runtime container required for generating an XR application is integrated internally, and the runtime container is used to provide a runtime environment during the generation or operation of the XR application. The content generator also includes: A 3D scene effect editing module, for responding to an XR application generation operation, loading a 3D scene static resource, and editing 3D scene dynamic information for the 3D scene static resource based on a runtime environment provided by the runtime container to obtain 3D scene resource data; A 3D application logic editing module, configured to edit application logic code for the 3D scene resource data based on a runtime environment provided by the runtime container, so as to obtain application content of the XR application; The application content and the runtime container form the XR application, and the runtime container in the XR application provides a runtime environment for the application content during the running of the XR application.

2. The content generator according to claim 1, It is characterized in that The runtime container includes: a framework layer, a runtime environment layer and a software library layer; The software library layer includes multiple software engines and provides second-class application programming interfaces API of the multiple software engines, so as to obtain the first-class API by encapsulating the second-class API; The framework layer is used to provide the first class API to application developers, so that the application developers can write application logic code of the XR application based on the first class API, and perceive target events during the generation or operation of the XR application, run at least one function entity of the first class API corresponding to the target event, and provide the target API encapsulated in the function entity to the runtime environment layer, where the target API belongs to the second class API; The runtime environment layer is used to call the corresponding software engine in the software library layer to respond to the target event according to the target API.

3. The content generator according to claim 2, It is characterized in that The software library layer is further used to: provide the second type of API to the runtime environment layer, encapsulate the first type of API based on the second type of API, and provide the first type of API to the framework layer, so that the framework layer provides the first type of API to application developers; or The software library layer is also used to provide the second type of API to the runtime environment layer; the runtime environment layer is also used to encapsulate the first type of API based on the second type of API, and provide the first type of API to the framework layer, so that the framework layer can provide the first type of API to application developers.

4. The content generator according to claim 2 or 3, It is characterized in that The 3D scene effect editing module is specifically used for: In response to the loading operation of the 3D scene static resource, the runtime container is started, and the 3D scene static resource is run in the runtime environment provided by the runtime container to obtain a 3D static picture; In response to the editing operation on the 3D static screen, editing 3D scene dynamic information for the 3D scene static resource to obtain first intermediate state resource data; In response to the scene effect viewing operation, the runtime container is re-invoked, and the runtime environment provided by the runtime container is The first intermediate state resource data is run in the environment to obtain a first intermediate state 3D picture; When the first intermediate 3D picture does not satisfy the scene effect, the 3D scene dynamic information is adjusted to obtain the scene effect, so as to obtain 3D scene resource data.

5. The content generator according to claim 2 or 3, It is characterized in that The 3D application logic editing module is specifically used for: In response to the loading operation of the 3D scene resource data, the runtime container is started, and the 3D scene resource data is run in the runtime environment provided by the runtime container to obtain a first intermediate 3D picture; In response to the editing operation of the application logic code, second intermediate resource data is generated according to the 3D scene resource data and the application logic code, and the runtime container is re-invoked to run the second intermediate resource data in the runtime environment provided by the runtime container to obtain a second intermediate 3D picture; When the second intermediate 3D image does not satisfy the interaction effect, the application logic code is adjusted to satisfy the interaction effect to obtain the application content of the XR application.

6. A method for editing an XR application, It is characterized in that Applied to a content generator, the content generator is integrated with a runtime container, the runtime container is used to provide a runtime environment during the generation and operation of the XR application, the method includes: Respond to XR application generation operations and load 3D scene static resources; Based on the runtime environment provided by the runtime container, adding 3D scene dynamic information to the 3D scene static resource to obtain 3D scene resource data; Based on the runtime environment provided by the runtime container, edit application logic code for the 3D scene resource data to obtain application content of the XR application; The application content and the runtime container form the XR application, and the runtime container in the XR application provides a runtime environment for the application content during the running of the XR application.

7. The method according to claim 6, It is characterized in that Based on the runtime environment provided by the runtime container, 3D scene dynamic information is added to the 3D scene static resource to obtain 3D scene resource data, including: In response to the loading operation of the 3D scene static resource, the runtime container is started, and the 3D scene static resource is run in the runtime environment provided by the runtime container to obtain a 3D static picture; In response to the editing operation on the 3D static screen, editing 3D scene dynamic information for the 3D scene static resource to obtain first intermediate state resource data; In response to the scene effect viewing operation, the runtime container is re-invoked, and the first intermediate state resource data is run in the runtime environment provided by the runtime container to obtain a first intermediate state 3D picture; When the first intermediate 3D picture does not satisfy the scene effect, the 3D scene dynamic information is adjusted to obtain the scene effect, so as to obtain 3D scene resource data.

8. The method according to claim 7, It is characterized in that The runtime container includes: a framework layer, a runtime environment layer and a software library layer; Running the 3D scene static resource or the first intermediate resource data in the runtime environment provided by the runtime container to obtain a 3D static picture or a first intermediate 3D picture includes: The framework layer senses a first loading event of the 3D scene static resource or a first editing event on the 3D static screen, and runs a function entity of at least one first-category API corresponding to the first loading event or the first editing event; when running to a first target API encapsulated in the function entity, the first target API is provided to the runtime environment layer, and the first target API belongs to the second-category API; The runtime environment layer calls the corresponding software engine in the software library layer to respond to the first loading event or the first editing event according to the first target API; The first type of API is obtained by encapsulating the second type of API, and the second type of API is the API corresponding to the software engine included in the software library layer.

9. The method according to any one of claims 6 to 7, It is characterized in that Based on the runtime environment provided by the runtime container, editing application logic code for the 3D scene resource data to obtain application content of the XR application includes: In response to the loading operation of the 3D scene resource data, the runtime container is started, and the 3D scene resource data is run in the runtime environment provided by the runtime container to obtain a first intermediate 3D picture; In response to the editing operation of the application logic code, second intermediate resource data is generated according to the 3D scene resource data and the application logic code, and the runtime container is re-invoked to run the second intermediate resource data in the runtime environment provided by the runtime container to obtain a second intermediate 3D picture; When the second intermediate 3D image does not satisfy the interaction effect, the application logic code is adjusted to satisfy the interaction effect to obtain the application content of the XR application.

10. The method according to claim 9, It is characterized in that The runtime container includes: a framework layer, a runtime environment layer and a software library layer; Running the 3D scene resource data or the second intermediate resource data in the runtime environment provided by the runtime container to obtain a first intermediate 3D picture or a second intermediate 3D picture includes: The framework layer senses a second loading event of the 3D scene resource data or a second editing event of the application logic code, and runs a function entity of at least one first-category API corresponding to the second loading event or the second editing event; when running to a second target API encapsulated in the function entity, the second target API is provided to the runtime environment layer, and the second target API belongs to the second-category API; The runtime environment layer calls the corresponding software engine in the software library layer to respond to the second loading event or the second editing event according to the second target API; The first type of API is obtained by encapsulating the second type of API, and the second type of API is the API corresponding to the software engine included in the software library layer.

11. An electronic device, It is characterized in that The invention comprises a memory and a processor, wherein the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to implement the steps in the method according to any one of claims 6 to 10.

12. A computer-readable storage medium storing a computer program, It is characterized in that When the computer program is executed by a processor, the processor is enabled to implement the steps of the method according to any one of claims 6 to 10.

Citation Information

Patent Citations

  • A VR content development platform and method

    CN109887097A

  • Cross-platform rendering method and device

    CN113181657A

  • Method for image rendering and related device

    CN116089056A

  • XR application editing method and device and storage medium

    CN116339737A

  • Non-Disruptively Moving A Storage Fleet Control Plane

    US20220317912A1