Development and operation method and device of augmented reality application, equipment and medium

Abstract

The invention discloses an augmented reality application development and operation method and device, equipment and a medium, and belongs to the field of data processing. The method comprises the following steps: acquiring a target AR application mode selected by a user in a plurality of AR application modes through a mode setting interface in an AR container; according to the target AR application mode, video stream data and 3D matrix data of an AR camera in the AR container are pushed to a 3D engine or pushed to the 3D engine and a native AR engine, AR content and 3D content are obtained, and the 3D engine comprises the native 3D engine in the AR container or a webpage 3D engine outside the AR container; according to the target AR application mode, at least two layers of views are constructed, the AR content and the 3D content are rendered on at least part of views in the at least two layers of views, a comprehensive view of the AR application of the target AR application mode is obtained, at least two layers of views corresponding to different AR application modes are different in structure, and views where the AR content and the 3D content are rendered are different. According to the embodiment of the invention, the flexibility of AR application development can be improved.

Description

Technical Field

[0001] The present application belongs to the field of data processing, and in particular, relates to a method, device, equipment and medium for developing and operating an augmented reality application. Background Art

[0002] With the continuous development of information technology, augmented reality (AR), which integrates virtual environment and display environment, has come into being. AR construction has been carried out in education, industry, commerce and other scenarios, allowing users to realize a variety of business functions through AR applications.

[0003] However, the construction of AR applications faces high technical challenges, and different scenarios have different requirements for AR applications. For example, some scenarios require higher rendering performance for AR applications, hoping to optimize the experience of motion tracking and 3D rendering; some scenarios require higher efficiency for AR applications, hoping that AR applications can be lighter and launched quickly. However, current AR application development tools can only meet a single requirement for AR applications, and it is difficult to support the development of AR applications with multiple different requirements, making the flexibility of AR application development very low. Summary of the invention

[0004] The embodiments of the present application provide a method, apparatus, device and medium for developing and operating an augmented reality application, which can improve the flexibility of AR application development.

[0005] In a first aspect, an embodiment of the present application provides a method for developing and running an augmented reality application, comprising: obtaining a target AR application mode selected by a user from multiple AR application modes through a mode setting interface in an augmented reality AR container; according to the target AR application mode, pushing video stream data and 3D matrix data provided by an AR camera in the AR container to a 3D engine, or, pushing to the 3D engine and a native AR engine in the AR container, to obtain AR content and 3D content, the 3D engine including a native 3D engine in the AR container or a web 3D engine outside the AR container; according to the target AR application mode, constructing at least two layers of views, rendering the AR content and the 3D content on at least part of the views in the at least two layers of views, to obtain a comprehensive view of the AR application of the target AR application mode, wherein the at least two layers of views corresponding to different AR application modes have different structures and the views in which the AR content and the 3D content are rendered are different.

[0006] In a second aspect, an embodiment of the present application provides a development and operation device for an augmented reality application, including: a mode determination module, used to obtain a target AR application mode selected by a user from multiple AR application modes through a mode setting interface in an augmented reality AR container; a container processing module, used to push video stream data and 3D matrix data provided by an AR camera in the AR container to a 3D engine according to the target AR application mode, or to push them to the 3D engine and a native AR engine in the AR container to obtain AR content and 3D content, the 3D engine including a native 3D engine in the AR container or a web 3D engine outside the AR container; a view display module, used to construct at least two layers of views according to the target AR application mode, render the AR content and the 3D content on at least part of the views in the at least two layers of views, and obtain a comprehensive view of the AR application of the target AR application mode, wherein the at least two layers of views corresponding to different AR application modes have different structures and the views where the AR content and the 3D content are rendered are different.

[0007] In a third aspect, an embodiment of the present application provides a device for developing and operating an augmented reality application, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the method for developing and operating the augmented reality application of the first aspect is implemented.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon, and when the computer program instructions are executed by a processor, the method for developing and operating an augmented reality application of the first aspect is implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method for developing and running an augmented reality application of the first aspect.

[0010] The embodiments of the present application provide a method, apparatus, device and medium for developing and operating an augmented reality application, which can obtain the target AR application mode selected by the user from a variety of AR application modes, select the corresponding data transmission strategy and data processing strategy according to the target application mode, and push the video stream data and 3D matrix data to the 3D engine, or push it to the native AR engine and 3D engine to obtain the AR content and 3D content required by the AR application. According to the target application mode, the corresponding view rendering strategy is selected to render the AR content and 3D content onto at least two layers of constructed views. By combining different data transmission strategies, data processing strategies, and view rendering strategies, the development performance of the AR application is matched with the development performance requirements of the AR application mode, and the development of AR applications with multiple AR application modes with different performances can be supported and compatible, thereby improving the flexibility of AR application development. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0012] Figure 1 A schematic diagram of a framework of an example of an AR container provided in an embodiment of the present application;

[0013] Figure 2 A flowchart of a method for developing and operating an augmented reality application provided in one embodiment of the present application;

[0014] Figure 3 A schematic diagram of an example of a view in a first AR application mode provided in an embodiment of the present application;

[0015] Figure 4 A schematic diagram of an example of a view in the second AR application mode provided in an embodiment of the present application;

[0016] Figure 5 A schematic diagram of an example of a performance optimization process of an AR application in a second AR application mode provided in an embodiment of the present application;

[0017] Figure 6 A schematic diagram of an example of a view in the third AR application mode provided in an embodiment of the present application;

[0018] Figure 7 A schematic diagram of an example of a performance optimization process of an AR application in a third AR application mode provided in an embodiment of the present application;

[0019] Figure 8 A schematic diagram of an example of a test view provided in an embodiment of the present application;

[0020] Fig. 9 A schematic diagram of the structure of an augmented reality application development and operation device provided in one embodiment of the present application;

[0021] Fig.10 A schematic diagram of the structure of a device for developing and running an augmented reality application provided in one embodiment of the present application. DETAILED DESCRIPTION

[0022] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are intended only to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by showing examples of the present application. It should be noted that the acquisition, storage, use, and processing of information and data in the embodiments of the present application are authorized by users or relevant agencies, in compliance with the relevant provisions of national laws and regulations.

[0023] With the continuous development of information technology, AR, which integrates virtual environment and display environment, has emerged. AR construction has been carried out in education, industry, commerce and other scenarios, allowing users to realize a variety of business functions through AR applications. However, there are high technical challenges in the construction of AR applications, and the requirements for AR applications in different scenarios are also quite different. For example, some scenarios have higher requirements for the rendering performance of AR applications, hoping to optimize the experience of motion tracking and 3D rendering; some scenarios have higher requirements for the efficiency of AR applications, hoping that AR applications can be lighter and go online quickly. However, the current AR application development tools can only meet the single requirement for AR applications, and it is difficult to support the development of AR applications with multiple different requirements, making the flexibility of AR application development very low.

[0024] The present application provides a development and operation method, device, equipment, medium and program product for augmented reality applications. Through different multi-rendering view modes corresponding to different AR application modes, the cropped native AR engine and the native 3D engine are connected, and the data flow and rendering are optimized through the native AR camera combined with the native engine and the web engine (i.e., the Web engine), and the core interface matching the rendering view is bridged for the web page to call. It can give full play to the native performance, that is, Native performance, support the construction of efficient AR motion tracking and high-quality 3D content rendering; it can also improve the operating efficiency of AR applications and ensure high-quality AR motion tracking and 3D content rendering; it can also support the rapid and smooth migration of existing AR applications to the higher-performance AR container of the embodiment of the present application; it enables developers to choose different AR application solutions according to different performance requirements, improve the flexibility of AR application development, improve the development efficiency of AR applications, and reduce the development cost of AR applications.

[0025] For ease of understanding, the structure of the AR container in the embodiment of the present application is first described here. Figure 1 A schematic diagram of an example of an AR container provided in an embodiment of the present application is shown in FIG. Figure 1As shown, the AR container 10 may include a basic engine 11 , a basic view 12 , and a business interface 13 .

[0026] The basic engine 11 may include a native AR engine 111, a native 3D engine 112, a native AR camera 113, an inertial measurement unit (IMU) 114, a traditional rendering interface 115, and a modern graphics interface 116. The basic engine can implement AR algorithms and 3D rendering, and constitutes the underlying capabilities supported by more than 10 AR application modes of the AR container. The rendering effect of the native AR engine 111 is better than that of the web AR engine. The rendering effect of the native 3D engine 112 is better than that of the web 3D engine. The traditional rendering interface 115 may include but is not limited to openGL ES (i.e., Open Graphics Library for Embedded Systems), which can be used to render 2D and 3D vector graphics. The modern graphics interface 116 may include but is not limited to Vulkan interface, Metal interface, etc., which can be used to render 2D and 3D vector graphics, and can provide closer access to hardware than the traditional rendering interface 115.

[0027] The basic view 12 can provide a native view 121 and a web page view 122. The use of different rendering views can be achieved through the combination and scheduling of different views in the basic view. The combination and scheduling of different views in the basic view can also match the data transmission and interaction method of the underlying basic engine. The basic view can form the basis for supporting multiple AR application modes.

[0028] The business interface 13 may include a native AR engine application programming interface (Application Programming Interface, API) 131, a native 3D engine API 132, other APIs 133 and a bridge framework 134. The business interface can bridge the core capabilities of the basic engine 11 to the script layer, so that the same native code can create AR applications on multiple terminals, realizing the cross-platform foundation of the AR container 10, and also enabling the script layer to build AR applications with high rendering quality.

[0029] The following describes the development and operation method, device, equipment, medium and program product of the augmented reality application provided by the present application.

[0030] In a first aspect, the present application provides a method for developing and operating an augmented reality application, which can be applied to scenarios in which developers develop AR applications. The method for developing and operating an augmented reality application can be specifically executed by an augmented reality application development and operation device, equipment, etc. Figure 2 A flowchart of a method for developing and operating an augmented reality application provided in one embodiment of the present application, such as Figure 2As shown, the method for developing and running the augmented reality application may include steps S201 to S203.

[0031] In step S201, a target AR application mode selected by a user from among multiple AR application modes is obtained through a mode setting interface in an augmented reality AR container.

[0032] Multiple AR application modes are AR application modes supported by the AR container. A mode setting interface may be set in the AR container to obtain the AR application mode selected by the user input. The target AR application mode includes the AR application mode selected by the user from multiple AR application modes. Different AR application modes correspond to different performance requirements of AR applications, and the performance requirements may include but are not limited to requirements for rendering effects, development efficiency, application migration, etc. In some examples, multiple AR application modes may include a first AR application mode, a second AR application mode, and a third AR application mode. The first AR application mode may be an AR application mode that requires extremely high rendering effects, the second application mode may be an application mode that requires high rendering effects and high development efficiency, and the third application mode may be an application mode that requires fast and smooth application migration effects. The first AR application mode may also be called a high-performance hybrid AR application, which can give full play to native performance, so that developers who are accustomed to using ordinary web development languages ​​such as H5 can also develop AR applications with high rendering quality through the development and operation method of augmented reality applications provided in the embodiments of the present application. The second AR application mode may also be referred to as a high-performance Web AR application mode, which has higher rendering performance and higher development efficiency than the existing Web AR application development method. It is suitable for a short development cycle and hopes to quickly build an AR application development scenario through a web page mode, i.e., a Web mode. The third AR application mode may also be referred to as a conventional general-purpose Web AR application mode, which can quickly and smoothly migrate existing AR applications to the AR container in the embodiment of the present application, quickly build an AR application ecosystem, and quickly reuse existing Web AR applications. The first AR application mode, the second AR application mode, and the third AR application mode enable users, i.e., developers, to choose different AR application models according to different development needs.

[0033] The basic view in the AR container combined with the basic engine can bridge the mode setting interface, which can be recorded as the setAppMode interface. The mode setting interface is used to set the identifier of the AR application mode and obtain the identifier of the target AR application mode selected by the user. For example, if the AR application mode may include a first AR application mode, a second AR application mode, and a third AR application mode, the identifier of the first AR application mode may be 01, the identifier of the second AR application mode may be 02, and the identifier of the third AR application mode may be 03. According to the identifier obtained by the mode setting interface, the target AR application mode selected by the user can be determined.

[0034] In step S202, according to the target AR application mode, the video stream data and 3D matrix data provided by the AR camera in the AR container are pushed to the 3D engine, or pushed to the 3D engine and the native AR engine in the AR container to obtain AR content and 3D content.

[0035] Different AR application modes correspond to different data transmission and data processing. For example, in different AR application modes, the receivers of video stream data and 3D matrix data may be different. The AR camera here is a native AR camera, and the video stream data is the stream data of the video captured by the AR camera. The data stream user constructs AR content. The 3D matrix data may include a view matrix and a pose matrix, which are used to construct 3D content.

[0036] The 3D engine includes a native 3D engine in an AR container or a web 3D engine (i.e., Web 3D engine) outside an AR container. The web 3D engine may be a general Web 3D engine enabled on the Web side, and the web 3D engine may have WebGL (i.e., Web Graphics Library) functionality. For example, some AR application modes will push video stream data and 3D matrix data directly to the native 3D engine, and some AR application modes will push video stream data to the web 3D engine through an interface of a script layer such as the JavaScript layer (referred to as the js layer).

[0037] Different AR application modes correspond to different data transmission and data processing methods, which can meet the performance development requirements of different AR application modes.

[0038] In step S203, at least two layers of views are constructed according to the target AR application mode, and AR content and 3D content are rendered on at least part of the views in the at least two layers of views to obtain a comprehensive view of the AR application of the target AR application mode.

[0039] The structures of at least two layers of views corresponding to different AR application modes are different, and the views in which the AR content and 3D content are rendered are different. The structure of at least two layers of views may include, but are not limited to, the types of views in at least two layers of views, and the positional relationship between views in at least two layers of views. The types of views in at least two layers of views corresponding to different AR application modes may be different; the positional relationship between at least two layers of views corresponding to different AR application modes may be different; in different AR application modes, the views in which the AR content is rendered may be different, and the views in which the 3D content is rendered may be different; it should be noted that if at least one of the types of views in at least two layers of views corresponding to two AR application modes, the positional relationship between at least two layers of views, the views in which the AR content is rendered, and the views in which the 3D content is rendered are different, they can be regarded as different AR application modes. The comprehensive view is the view obtained by superimposing at least two layers of views.

[0040] By corresponding different AR application modes to different viewing modes, the performance development requirements of different AR application modes can be met.

[0041] In an embodiment of the present application, the target AR application mode selected by the user from a variety of AR application modes can be obtained, and the corresponding data transmission strategy and data processing strategy can be selected according to the target application mode, and the video stream data and 3D matrix data can be pushed to the 3D engine, or pushed to the native AR engine and the 3D engine to obtain the AR content and 3D content required for the AR application. According to the target application mode, the corresponding view rendering strategy is selected to render the AR content and 3D content to at least two layers of constructed views. Through the combination of different data transmission strategies, data processing strategies, and view rendering strategies, the development performance of the AR application is matched with the development performance requirements of the AR application mode, and the development of AR applications with multiple AR application modes of different performance can be supported and compatible, thereby improving the flexibility and compatibility of AR application development, improving the development efficiency of AR applications, and reducing the development cost of AR applications.

[0042] In some embodiments, the target AR application mode may include a first AR application mode. The above step S202 can be specifically refined as follows: pushing the video stream data to the native 3D engine, setting the sky box as AR content through the native 3D engine; pushing the 3D matrix data to the native 3D engine, rendering through the native 3D engine, and obtaining 3D content. For the first AR application mode, after initializing the native AR engine and the native 3D engine, instead of pushing the video stream data and the 3D matrix data to the script layer such as the js layer, and then pushing the video stream data and the 3D matrix data to the native 3D engine through the script layer using the interface, the video stream data and the 3D matrix data are directly pushed to the native 3D engine. The native 3D engine uses the sky box to set the background image to obtain AR content. The native 3D engine can render according to the 3D matrix data such as the pose matrix provided by the established AR session, i.e., ARSession, to obtain 3D content including a 3D model. In the first AR application mode, by directly pushing the video stream data and the 3D matrix data to the native 3D engine, the complex transmission of a large amount of data can be avoided, and the time required for data transmission can be shortened.

[0043] In the case where the target AR application mode includes the first AR application mode, the above step S203 can be specifically refined as follows: create and add a web view on the initial view, the web view is used to call the functional interface in the AR container in response to the interactive operation; create and add a native view on the web view; render the AR content and 3D content on the native view to obtain a comprehensive view. The user can interact with the AR application through interactive operations, and the web view can call the functional interface in the AR container in response to the interactive operation to implement the function corresponding to the interactive operation. In the first AR application mode, the native view is located above the web view, and the web view is located above the initial view. Both AR content and 3D view are displayed in the native view. For example, Figure 3 A schematic diagram of an example of a view in the first AR application mode provided in an embodiment of the present application, such as Figure 3 As shown, the native view can be implemented as GLSurfaceview31, the web view can be implemented as Webview 32, the initial view is View 33, a Webview 32 is created and added above View 33, a GLSurfaceview 31 is created and added above Webview 32, and both AR content and 3D content are rendered in GLSurfaceview 31. In the case where the user interacts with the AR application through interactive input in the future, the functional interface of the script layer in the AR container can also be implemented through interactive events to control the interaction of the 3D model in the 3D content on the native view. In some examples, the interactive information obtained by the web view in at least two views in response to the user's interactive input can be obtained, the collision detection function is called, and the interactive result information is determined according to the interactive information. The native 3D engine can directly obtain the 3D matrix data, and the native 3D engine uses the interactive result information and 3D matrix data to update the 3D content, and the shape and position of the 3D model can be updated, which is not limited here. The collision detection function can detect whether the ray emitted from the position indicated by the user's interactive input intersects or collides with the point or area in the collected real-world image, thereby obtaining the interactive result information, which can indicate whether the ray emitted from the position indicated by the interactive input intersects or collides with the point or area in the collected real-world image. The native 3D engine can transform the posture of the 3D model in the 3D content according to the interactive result information and 3D matrix data, thereby updating the 3D content.

[0044] In some embodiments, the target AR application mode includes a second AR application mode. The above step S202 can be specifically refined as follows: push the video stream to the native AR engine through the script layer, convert the video stream into texture information and update it to the bound texture object; obtain the texture object, and obtain AR content according to the texture object; push the 3D matrix data to the web 3D engine, set the background color of the renderer of the web 3D engine and the background color of the web application to transparent, and render in a loop according to the preset frame frequency to obtain 3D content. The video stream data can be converted into texture information that can be processed by the traditional rendering interface through texture conversion functions such as SurfaceTexture, and the texture information is updated to the bound texture object. The traditional rendering interface can obtain the texture object according to the pre-generated texture object identifier, and render the texture information in the texture object to obtain AR content. The AR content can be implemented as frame data, and the AR content can be displayed on the corresponding view through an interface library such as EGL (i.e., Embedded System Graphics Library). At the same time, a general web 3D engine can be opened on the web side, and the renderer and background color and the background color of the web application are all set to transparent color. The web application can be an application corresponding to the web view, and the web application can include an H5 application. The preset frame frequency can be obtained based on the frame frequency supported by the AR camera in the AR container and the frame frequency supported by the web view. For example, the preset frame frequency can be 30fps, where fps is the number of frames per second, that is, 3D matrix data is obtained every 30 seconds to obtain the position and posture of the 3D model in the 3D content, thereby cyclically updating the 3D model in the 3D content. In some examples, the 3D matrix data includes a pose matrix, which can represent the position and posture of the 3D model in the 3D content. The interactive information obtained by the web view in at least two layers of view in response to the user's interactive input can be obtained; the collision detection function is called to determine the interactive result information according to the interactive information; the 3D matrix data is obtained through the matrix data interface in the AR container, and the interactive result information and the 3D matrix data are used to update the 3D content through the web 3D engine. The web view such as webview can capture the user's interactive events such as touch events on the web page, and the interactive events are triggered by the user's interactive input. The implementation of the collision detection function can refer to the relevant description in the above embodiment, which will not be repeated here. The matrix data interface can be set in the script layer such as the js layer. The matrix data interface can use the getMatrix interface to obtain 3D matrix data through the matrix data interface and push it to the web 3D engine, namely the Web 3D engine. The 3D model can be updated according to the interactive result information and the 3D matrix data through the matrix update algorithm of the web 3D engine, thereby updating the 3D content including the 3D model. The updated 3D content is displayed on the corresponding view to realize the update of the content of the display interface of the AR application.

[0045] In the case where the target AR application mode includes the second AR application mode, the above step S203 can be specifically refined as follows: create and add a web view on the initial view, the web view is used to call the functional interface in the AR container in response to the interactive operation; create and add a native view under the web view; render the AR content on the native view, and render the 3D content on the web view to obtain a comprehensive view. The web view can call the functional interface in the AR container in response to the interactive operation to implement the function corresponding to the interactive operation. In the second AR application mode, the web view is above the native view, and the initial view is above the native view. AR content is displayed in the native view, and 3D content is displayed in the web view. For example, Figure 4 A schematic diagram of an example of a view in the second AR application mode provided in an embodiment of the present application, such as Figure 4 As shown, the native view can be implemented as GLSurfaceview31, the web view can be implemented as Webview 32, the initial view is View 33, a Webview 32 is created and added above View 33, a GLSurfaceview31 is created and added below Webview 32, the AR content is rendered in GLSurfaceview 31, and the 3D content is rendered in Webview 32.

[0046] For ease of understanding, the performance optimization process of the AR application in the second AR application mode is described below with reference to the accompanying drawings. Figure 5 A schematic diagram of an example of a performance optimization process of an AR application in a second AR application mode provided in an embodiment of the present application is shown as follows: Figure 5 As shown, in the java layer, the AR camera transmits the video stream data to SurfaceTexture, which converts the video stream data into texture information and binds the texture information to the texture object. The texture object has a texture object id, and the texture information can be bound to the texture object id, which is equivalent to updating the texture information to the bound texture object. The texture object can be implemented by the C++ layer, and the C++ layer can also implement OpenGL functions. The texture object id can be pre-generated by OpenGL. The texture object id has a corresponding relationship with the texture object, and the texture object id can identify the texture object. OpenGL obtains the texture information in the texture object according to the texture object id, thereby obtaining the AR content, and then renders it to the GLSurfaceview for display. The AR camera transmits the 3D matrix data to the js layer. The js layer can transform and update the 3D model according to the 3D matrix data through the WebGL function in the web 3D engine to obtain the 3D content, render it to the Webview frame by frame, and display it on the Canvas.

[0047] In some embodiments, the target AR application mode includes a third AR application mode. The above step S202 can be specifically refined as follows: according to the preset frame frequency, the video stream data is pushed to the web 3D engine through the video stream acquisition interface in the AR container, and the texture information corresponding to the video stream data is bound to the context of the web renderer of the web 3D engine to obtain AR content; according to the preset frame frequency, the 3D matrix data is obtained through the matrix data interface in the AR container, and the 3D matrix data is used to transform and update to obtain 3D content. Under the third AR application model where the performance requirement is to quickly and smoothly migrate the existing AR application, only the session establishment interface, i.e., the createSession interface, the video stream acquisition interface, i.e., the getBuffer interface, and the matrix data interface, i.e., the getMatrix interface, can be applied. When the AR session is created, the resolution of the AR camera can be appropriately reduced to reduce the amount of data transmission. The ratio of reducing the resolution can be set according to the scene, requirements, experience, etc., and is not limited here. For example, the resolution can be reduced by 2 / 3. After the AR session is created and motion tracking is turned on, the AR camera will continuously generate video stream data and 3D matrix data. A web 3D engine can be created, such as creating a general Web 3D engine based on WebGL. For the content of the preset frame frequency, please refer to the relevant description in the above embodiment, which will not be repeated here. The video stream data can be obtained from the AR camera through the video stream acquisition interface of the script layer such as the js layer, such as the getBuffer interface, according to the preset frame frequency, and the video stream data can be converted into texture information. The texture information can be implemented as but not limited to yuv texture information. When the posture of the 3D model changes, the 3D matrix data can be obtained through the matrix data interface of the script layer such as the js layer, such as the getMatrix interface, and the posture of the 3D model can be transformed through the update method of the web 3D engine to obtain 3D content.

[0048] In some examples, in the process of pushing video stream data to the web 3D engine through the video stream acquisition interface in the AR container, the feature point tracking function can be used to determine whether the adjacent frame images represented by the video stream data of the AR camera are similar. If the adjacent frame images represented by the video stream data of the AR camera are similar, the video stream data of the previous frame image can be reused, and the video stream data of each frame image does not need to be pushed to the web 3D engine, thereby reducing the frequency of data transmission and reducing the time spent on data transmission. Specifically, the feature point tracking function is used to record the position of the feature points in each frame image represented by the video stream data; if the number of feature points and the position of the feature points in the current frame image and the number of feature points and the position of the feature points in the previous frame image meet the reuse conditions, then the push reuse flag is pushed to the script layer in the AR container, so that the script layer skips the step of pushing the current frame image to the web 3D engine, and the web 3D engine reuses the texture information corresponding to the video stream data of the previous frame image. The position of the feature point may include the display position of the feature point on the screen of the device. The reuse conditions may include but are not limited to the difference between the number of feature points in the current frame image and the number of feature points in the previous frame image being within the number error range, and the offset between the position of the feature points in the current frame image and the position of the feature points in the previous frame image being within the offset error range. The reuse identifier may be recorded as reuse, which is used to indicate that the number of feature points in the current frame image, the position of the feature points, and the number of feature points and the position of the feature points in the previous frame image meet the reuse conditions. If the script layer receives the reuse identifier, the video stream data corresponding to the current frame image will not be transmitted to the js layer, nor will it be transmitted to the web 3D engine. The web 3D engine can directly reuse the texture information corresponding to the video stream data of the previous frame image as the texture information of the current frame image. In other words, if the change between the feature points in the current frame image and the feature points in the previous frame image is small, it means that the change between the two frames is small, and the video stream data of the previous frame image can be reused as the video stream data of the current frame image for subsequent processing. The web 3D engine can reuse the video stream data of the previous frame image cached by the script layer, convert the video stream data of the previous frame image into texture information, and then process the texture information; the web 3D engine can also directly reuse the texture information obtained by converting the video stream data of the previous frame image. In some examples, the resolution of the video stream data can also be reduced, for example, the video stream data can be processed into video stream data with a screen resolution of 0.4K.

[0049] In the case where the target AR application mode includes the third AR application mode, the above step S203 can be specifically refined as follows: create and add a web view on the initial view, the web view is used to call the functional interface in the AR container in response to the interactive operation; render the AR content and the 3D content on the web view to obtain a comprehensive view. The web view can call the functional interface in the AR container in response to the interactive operation to implement the function corresponding to the interactive operation. In the third AR application mode, the web view is above the initial view. Both the AR content and the 3D content are displayed in the web view. For example, Figure 6 A schematic diagram of an example of a view in the third AR application mode provided in an embodiment of the present application, such as Figure 6 As shown, the web page view can be implemented as Webview 32, the initial view is View 33, a Webview 32 is created and added above View 33, and both the AR content and the 3D content are rendered in Webview32.

[0050] For ease of understanding, the performance optimization process of the AR application in the third AR application mode is described below with reference to the accompanying drawings. Figure 7 A schematic diagram of an example of a performance optimization process of an AR application in the third AR application mode provided in an embodiment of the present application is shown as follows: Figure 7 As shown, in the java layer, the getBuffer interface is called according to the preset frame frequency to obtain video stream data from the AR camera, and the video stream data is transmitted to the script layer such as the js layer. The video stream data is converted into yuv texture information through yuvTexture, and bound to the context of the WebGL renderer of the web 3D engine, that is, render, to obtain AR content through WebGL. The AR camera outputs 3D matrix data, and the 3D matrix data is transmitted to the js layer. According to the 3D matrix data, it is transformed and updated through WebGL to obtain 3D content. The AR content and 3D content are rendered to the web view frame by frame and displayed on the Canvas.

[0051] In some embodiments, the basic engine in the AR container can be set as a lightweight engine. The AR container may include functional modules, which include extended functional modules and basic functional modules. The basic functional modules include functional modules of the native AR engine and functional modules of the native 3D engine. The basic functional modules include functional modules that can realize the basic functions of AR applications. The extended functional modules include functional modules that can realize functions other than the basic functions of AR applications. For example, the basic functional modules may include functional modules such as posture estimation, environmental perception, feature point extraction, tracking, sensor fusion, environmental mapping, and model construction, i.e., Mapping, which are abstracted from the AR algorithm. These functional modules are functional modules of the native AR engine; the basic functional modules may include functional modules such as graphics rendering, rendering pipeline, resource loading, user input, and collision detection obtained by splitting the 3D engine. These functional modules are functional modules of the native 3D engine; the extended functional modules may include atomic algorithm functional modules split from the target recognition algorithm, such as QR code recognition, 2D recognition, 3D recognition, face and body recognition, and other functional modules; the extended functional modules may also include, but are not limited to, complex rigid body dynamics in the physics engine, complex high dynamic range (HDR), ambient light, dynamic blur processing, and editor-dependent functions. In some examples, the basic function module may only support the Graphics Library Transmission Format (GLTF), and other 3D model formats may be used as extended function modules. It is possible to determine whether to download extended function modules and which extended function modules to download based on the functions that the host device of the AR application can support and the performance of the AR application, so as to ensure the performance requirements of the AR application. The interface design of the AR container can also be simplified as much as possible. For example, the AR container may include interfaces related to AR functions and interfaces related to 3D functions; the interfaces related to the AR function may include but are not limited to the interface for initializing the AR session, namely the createSession interface, the video stream data acquisition interface, namely the getBuffer interface, and the matrix data interface, namely the getMatrix interface; the interfaces related to the 3D function may include the native 3D engine initialization interface, namely the init3DEngine interface, the model addition interface, namely the addModle interface, and other basic 3D JS APIs; the AR container does not have interfaces such as rawData that involve large amounts of data transmission except the video stream data acquisition interface getBuffer. During the actual operation of the AR application, the script code of each functional interface is run in a native script engine such as a native js engine. There is no need to introduce additional heavy js engines such as V8, which can further make the basic engine lightweight.

[0052] Specifically, during the process of an AR application running on a host device, a test view can be generated outside the display window of the AR application using a basic functional module; based on the performance of the test view and the device information of the host device, a target rendering capability level is obtained; a target functional module corresponding to the target rendering capability level is determined in a preset rendering capability module relationship, and the rendering capability module relationship includes a correspondence between the rendering capability level and the functional module; the target functional module is merged with the original native code of the AR application to obtain the target native code, and the target native code runs in an AR container. The original native code of the AR application can be run using the basic functional module first, and a test view can also be generated while generating the display window of the AR application. The test view is invisible to the user outside the display window of the AR application, that is, the test view is imperceptible to the user. For example, Figure 8 A schematic diagram of an example of a test view provided in an embodiment of the present application, such as Figure 8 As shown, the host device displays a display window 41, and the display area of ​​the host device can only display the display window 41. The test view 42 is outside the display window and outside the display area. The performance of the test view generated by the same basic function module using the same basic function module of AR applications with different performance may be different. For example, the frame frequency of the test view generated by the same basic function module using the same basic function module of AR applications with different performance is different. The performance of the test view can reflect the performance of the AR application. The device information of the host device may include but is not limited to the central processing unit model, the graphics processing unit model, the memory size, the gyroscope model, the accelerometer model, the magnetometer model, etc. The device information of the host device can reflect the AR performance that the host device can support. Combining the performance of the AR application and the AR performance that the host device can support, the rendering capability level of the host device running the AR application, that is, the target rendering capability level, can be determined. In some examples, a first rendering capability score can be obtained according to the performance of the test view; a second rendering capability score can be obtained according to the device information of the host device; the first rendering capability score and the second rendering capability score are weighted to obtain a rendering capability score; and the target rendering capability level is determined according to the rendering capability score. The first rendering capability score can reflect the performance of the AR application. The higher the first rendering capability score, the higher the performance of the AR application. The second rendering capability score may reflect the AR performance supported by the host device. The higher the second rendering capability score, the higher the AR performance supported by the host device. The higher the rendering capability score, the higher the rendering capability level. The corresponding relationship between the rendering capability score and the rendering capability level may be pre-set, so as to determine the target rendering capability level according to the calculated rendering capability score.

[0053] The rendering capability module relationship is a correspondence between pre-matched rendering capability levels and functional modules, and different rendering capability levels correspond to different functional modules in the rendering capability module relationship. In some examples, the functional modules recorded in the rendering capability module relationship may include basic functional modules and extended functional modules. For example, the rendering capability level A1 is lower than the rendering capability level A2, and the functional modules corresponding to the rendering capability level A1 include all basic functional modules, and the functional modules corresponding to the rendering capability level A2 include all basic functional modules and the ambient light module, and the ambient light module is an extended functional module. In other examples, the functional modules recorded in the rendering capability module relationship include extended functional modules, and all rendering capability levels can use all basic functional modules by default. For example, the functional module corresponding to the rendering capability level A1 is none, that is, it does not include an extended capability module, and the functional module corresponding to the rendering capability level A2 includes an ambient light module, but the host devices of the rendering capability levels A1 and A2 can use all basic functional modules. The target functional module includes the functional module corresponding to the target rendering capability level in the rendering capability module relationship. The target functional module is merged with the original native code, that is, the extended functional module is merged into the native code to extend the native code function.

[0054] In some examples, when the user selects the target AR application mode, the device share information can be pushed to the user. The device share information can be used to characterize the performance of various devices in terms of rendering performance. The user can preliminarily understand the performance of the host device used in terms of rendering performance based on the device share information, so as to explain to the user the risks of calling various functional modules. If there is a call to a functional module that does not match the target rendering capability level in the native code developed by the user, the call to the functional module that does not match the target rendering capability level in the native code can be replaced with a call to a functional module that matches the target rendering capability level, that is, the function module call is downgraded.

[0055] The basic engine in the lightweight AR container and the dynamic download of extended function modules according to actual conditions can reduce the size of the AR application package and solve the problem of large package size increase caused by the native engine.

[0056] A second aspect of the present application provides a device for developing and running an augmented reality application. Fig. 9 A schematic diagram of the structure of an augmented reality application development and operation device provided in an embodiment of the present application, such as Fig. 9 As shown, the development and operation device 500 for the augmented reality application may include a mode determination module 501 , a container processing module 502 and a view display module 503 .

[0057] The mode determination module 501 may be used to obtain a target AR application mode selected by a user from among multiple AR application modes through a mode setting interface in an augmented reality AR container.

[0058] The container processing module 502 may be used to push the video stream data and 3D matrix data provided by the AR camera in the AR container to the 3D engine, or to push the video stream data and 3D matrix data to the 3D engine and the native AR engine in the AR container according to the target AR application mode, so as to obtain AR content and 3D content. The 3D engine includes the native 3D engine in the AR container or the web 3D engine outside the AR container.

[0059] The view display module 503 may be used to construct at least two layers of views according to the target AR application mode, render AR content and 3D content on at least part of the views in the at least two layers of views, and obtain a comprehensive view of the AR application of the target AR application mode.

[0060] The structures of at least two layers of views corresponding to different AR application modes are different, and the views in which the AR content and the 3D content are rendered are different.

[0061] In some embodiments, the target AR application mode includes the first AR application mode. The view display module 503 can be specifically used to: create and add a web page view on the initial view, the web page view is used to call the function interface in the AR container in response to the interactive operation; create and add a native view on the web page view; render the AR content and the 3D content on the native view to obtain a comprehensive view.

[0062] In some embodiments, the target AR application mode includes a second AR application mode. The view display module 503 can be specifically used to: create and add a web page view on the initial view, the web page view is used to call the function interface in the AR container in response to the interactive operation; create and add a native view under the web page view; render the AR content on the native view, and render the 3D content on the web page view to obtain a comprehensive view.

[0063] In some embodiments, the target AR application mode includes a third AR application mode. The view display module 503 can be specifically used to: create and add a web page view on the initial view, the web page view is used to call the function interface in the AR container in response to the interactive operation; render the AR content and the 3D content on the web page view to obtain a comprehensive view.

[0064] In some embodiments, the target AR application mode includes a first AR application mode. The container processing module 502 can be specifically used to: push the video stream data to the native 3D engine, set the sky box as AR content through the native 3D engine; push the 3D matrix data to the native 3D engine, and render through the native 3D engine to obtain 3D content.

[0065] In some embodiments, the target AR application mode includes a second AR application mode. The container processing module 502 can be specifically used to: push the video stream to the native AR engine through the script layer, convert the video stream into texture information and update it to the bound texture object; obtain the texture object, and obtain AR content according to the texture object; push the 3D matrix data to the web 3D engine, set the background color of the renderer of the web 3D engine and the background color of the web application to transparent, and render in a loop according to the preset frame frequency to obtain 3D content.

[0066] In some examples, the target AR application mode includes a second AR application mode, and the 3D matrix data includes a pose matrix. The container processing module 502 can also be used to: obtain interactive information obtained by the web page view in at least two layers of views in response to the user's interactive input; call the collision detection function to determine the interactive result information according to the interactive information; obtain the 3D matrix data through the matrix data interface in the AR container, and update the 3D content through the web page 3D engine using the interactive result information and the 3D matrix data.

[0067] In some embodiments, the target AR application mode includes a third AR application mode. The container processing module 502 can be specifically used to: push the video stream data to the web 3D engine through the video stream acquisition interface in the AR container according to the preset frame frequency, bind the texture information corresponding to the video stream data to the context of the web renderer of the web 3D engine, and obtain AR content; obtain 3D matrix data through the matrix data interface in the AR container according to the preset frame frequency, and use the 3D matrix data to perform transformation and update to obtain 3D content.

[0068] In some examples, the container processing module 502 can be specifically used to: utilize the feature point tracking function to record the positions of feature points in each frame image represented by the video stream data; if the number of feature points in the current frame image, the positions of feature points and the number of feature points in the previous frame image, the positions of feature points meet the reuse conditions, then push a reuse identifier to the script layer in the AR container, so that the script layer skips the step of pushing the current frame image to the web 3D engine, and enables the web 3D engine to reuse the texture information corresponding to the video stream data of the previous frame image.

[0069] In some embodiments, the AR container includes a functional module, and the functional module includes an extended functional module and a basic functional module, and the basic functional module includes a functional module of a native AR engine and a functional module of a native 3D engine. The development and operation device 500 of the augmented reality application may also include a dynamic functional module. The dynamic functional module can be used to: generate a test view outside the display window of the AR application using the basic functional module during the operation of the AR application on the host device; obtain the target rendering capability level based on the performance of the test view and the device information of the host device; determine the target functional module corresponding to the target rendering capability level in the preset rendering capability module relationship, and the rendering capability module relationship includes the corresponding relationship between the rendering capability level and the functional module; merge the target functional module with the original native code of the AR application to obtain the target native code, and the target native code runs in the AR container.

[0070] It should be noted that the augmented reality application development and operation device 500 is a device corresponding to the above-mentioned augmented reality application development and operation method. All implementation methods in the above-mentioned method embodiment are applicable to the embodiment of the device and can achieve the same technical effect, which will not be repeated here.

[0071] The third aspect of the present application also provides a device for developing and running augmented reality applications. Fig.10 A schematic diagram of the structure of a development and operation device for an augmented reality application provided in an embodiment of the present application, such as Fig.10 As shown, an augmented reality application development and execution device 600 includes a memory 601 , a processor 602 , and a computer program stored in the memory 601 and executable on the processor 602 .

[0072] In some examples, the processor 602 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0073] The memory 601 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 601 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these.

[0074] The memory 601 may include a read-only memory (ROM), a random access memory (RAM), a disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Therefore, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the development and operation method of the augmented reality application in the embodiment of the present application.

[0075] The processor 602 runs a computer program corresponding to the executable program code by reading the executable program code stored in the memory 601 , so as to implement the development and operation method of the augmented reality application in the above embodiment.

[0076] In some examples, the augmented reality application development and operation device 600 may further include a communication interface 603 and a bus 604. Fig.10 As shown, the memory 601, the processor 602, and the communication interface 603 are connected via a bus 604 and communicate with each other.

[0077] The communication interface 603 is mainly used to realize the communication between the modules, devices, units and / or equipment in the embodiment of the present application. The communication interface 603 can also be used to access input devices and / or output devices.

[0078] The bus 604 includes hardware, software, or both, and couples the components of the augmented reality application development and operation device 600 to each other. By way of example and not limitation, the bus 604 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a Memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-E) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, the bus 604 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.

[0079] The fourth aspect of the present application also provides a computer-readable storage medium, on which a computer program instruction is stored. When the computer program instruction is executed by a processor, the development and operation method of the augmented reality application in the above-mentioned embodiment can be implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. Among them, the above-mentioned computer-readable storage medium may include a non-transitory computer-readable storage medium, such as a read-only memory (Read-Only Memory, referred to as ROM), a random access memory (Random Access Memory, referred to as RAM), a disk or an optical disk, etc., which is not limited here.

[0080] In a fifth aspect, the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the development and operation method of the augmented reality application in the above-mentioned embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0081] It should be clear that each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. For device embodiments, equipment embodiments, computer-readable storage medium embodiments, and computer program product embodiments, the relevant parts can refer to the description part of the method embodiment. The present application is not limited to the specific steps and structures described above and shown in the figures. Those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application. In addition, for the sake of brevity, a detailed description of known method technologies is omitted here.

[0082] The above reference is according to the method of the embodiment of the present application, the flow chart of the device (system) and the computer program product and / or the block diagram described various aspects of the present application.It should be understood that each square box in the flow chart and / or the block diagram and the combination of each square box in the flow chart and / or the block diagram can be realized by computer program instructions.These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the realization of the function / action specified in one or more square boxes of the flow chart and / or the block diagram.Such a processor can be but is not limited to a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit.It can also be understood that each square box in the block diagram and / or the flow chart and the combination of the square boxes in the block diagram and / or the flow chart can also be realized by the dedicated hardware that performs the specified function or action, or can be realized by the combination of dedicated hardware and computer instructions.

[0083] Those skilled in the art should understand that the above embodiments are exemplary rather than restrictive. Different technical features appearing in different embodiments can be combined to achieve beneficial effects. Based on the study of the drawings, the specification and the claims, those skilled in the art should be able to understand and implement other variations of the disclosed embodiments. In the claims, the term "comprising" does not exclude other devices or steps; the quantifier "one" does not exclude multiple; the terms "first" and "second" are used to indicate names rather than to indicate any specific order. Any figure marks in the claims should not be understood as limiting the scope of protection. The functions of multiple parts appearing in the claims can be implemented by a single hardware or software module. The appearance of certain technical features in different dependent claims does not mean that these technical features cannot be combined to achieve beneficial effects.

Claims

1. A method for developing and operating an augmented reality application, characterized in that: include: Obtain the target AR application mode selected by the user from multiple AR application modes through the mode setting interface in the augmented reality AR container; According to the target AR application mode, the video stream data and the 3D matrix data provided by the AR camera in the AR container are pushed to the 3D engine, or pushed to the 3D engine and the native AR engine in the AR container, to obtain AR content and 3D content, wherein the 3D engine includes the native 3D engine in the AR container or the webpage 3D engine outside the AR container; According to the target AR application mode, at least two layers of views are constructed, and the AR content and the 3D content are rendered on at least part of the views in the at least two layers of views to obtain a comprehensive view of the AR application of the target AR application mode. The at least two layers of views corresponding to different AR application modes have different structures and the views in which the AR content and the 3D content are rendered are different.

2. The method according to claim 1, characterized in that: The target AR application mode includes a first AR application mode; The step of constructing at least two layers of views according to the target AR application mode, rendering the AR content and the 3D content on at least a portion of the views in the at least two layers of views, and obtaining a comprehensive view of the AR application of the target AR application mode includes: Creating and adding a web page view on the initial view, where the web page view is used to call a function interface in the AR container in response to an interaction operation; Create and add a native view on the web view; The AR content and the 3D content are rendered on the native view to obtain the integrated view.

3. The method according to claim 1, characterized in that: The target AR application mode includes a second AR application mode; The step of constructing at least two layers of views according to the target AR application mode, rendering the AR content and the 3D content on at least a portion of the views in the at least two layers of views, and obtaining a comprehensive view of the AR application of the target AR application mode includes: Creating and adding a web page view on the initial view, where the web page view is used to call a function interface in the AR container in response to an interaction operation; Create and add a native view under the web view; The AR content is rendered on the native view, and the 3D content is rendered on the web page view to obtain the integrated view.

4. The method according to claim 1, characterized in that: The target AR application mode includes a third AR application mode; The step of constructing at least two layers of views according to the target AR application mode, rendering the AR content and the 3D content on at least a portion of the views in the at least two layers of views, and obtaining a comprehensive view of the AR application of the target AR application mode includes: Creating and adding a web page view on the initial view, where the web page view is used to call a function interface in the AR container in response to an interaction operation; The AR content and the 3D content are rendered on the web page view to obtain the integrated view.

5. The method according to claim 1, characterized in that The target AR application mode includes a first AR application mode; The step of pushing the video stream data and the 3D matrix data provided by the AR camera in the AR container to the 3D engine according to the target AR application mode, or pushing the video stream data and the 3D matrix data to the 3D engine and the native AR engine in the AR container to obtain the AR content and the 3D content includes: Pushing the video stream data to the native 3D engine, and setting the sky box as the AR content through the native 3D engine; The 3D matrix data is pushed to the native 3D engine, and the native 3D engine is used for rendering to obtain the 3D content.

6. The method according to claim 1, characterized in that The target AR application mode includes a second AR application mode; The step of pushing the video stream data and the 3D matrix data provided by the AR camera in the AR container to the 3D engine according to the target AR application mode, or pushing the video stream data and the 3D matrix data to the 3D engine and the native AR engine in the AR container to obtain the AR content and the 3D content includes: Pushing the video stream to the native AR engine through the script layer, converting the video stream into texture information and updating it into the bound texture object; Acquire the texture object, and obtain the AR content according to the texture object; The 3D matrix data is pushed to the web page 3D engine, the background color of the renderer of the web page 3D engine and the background color of the web page application are set to be transparent, and the rendering is cyclically performed according to a preset frame frequency to obtain the 3D content.

7. The method according to claim 1 or 6, characterized in that: The target AR application mode includes a second AR application mode, and the 3D matrix data includes a pose matrix; The method further comprises: Acquire interactive information obtained by the web page view in the at least two layers of views in response to the user's interactive input; Calling a collision detection function to determine interaction result information based on the interaction information; The 3D matrix data is obtained through the matrix data interface in the AR container, and the 3D content is updated through the web 3D engine using the interaction result information and the 3D matrix data.

8. The method according to claim 1, characterized in that The target AR application mode includes a third AR application mode; The step of pushing the video stream data and the 3D matrix data provided by the AR camera in the AR container to the 3D engine according to the target AR application mode, or pushing the video stream data and the 3D matrix data to the 3D engine and the native AR engine in the AR container to obtain the AR content and the 3D content includes: According to a preset frame frequency, the video stream data is pushed to the web 3D engine through the video stream acquisition interface in the AR container, and texture information corresponding to the video stream data is bound to the context of the web renderer of the web 3D engine to obtain the AR content; According to the preset frame frequency, the 3D matrix data is obtained through the matrix data interface in the AR container, and the 3D matrix data is used to perform transformation and update to obtain the 3D content.

9. The method according to claim 8, characterized in that Also includes: Using a feature point tracking function, recording the position of feature points in each frame of the image represented by the video stream data; If the number and positions of feature points in the current frame image and the number and positions of feature points in the previous frame image meet the reuse conditions, a reuse flag is pushed to the script layer in the AR container so that the script layer skips the step of pushing the current frame image to the web 3D engine and enables the web 3D engine to reuse the texture information corresponding to the video stream data of the previous frame image.

10. The method according to claim 1, characterized in that The AR container includes a functional module, the functional module includes an extended functional module and a basic functional module, and the basic functional module includes a functional module of the native AR engine and a functional module of the native 3D engine; The method further comprises: During the running of the AR application on the host device, using the basic function module to generate a test view outside the display window of the AR application; Based on the performance of the test view and the device information of the host device, obtaining a target rendering capability level; Determining a target functional module corresponding to the target rendering capability level in a preset rendering capability module relationship, wherein the rendering capability module relationship includes a corresponding relationship between the rendering capability level and the functional module; The target function module is merged with the original native code of the AR application to obtain the target native code, and the target native code is run in the AR container.

11. A development and operation device for augmented reality applications, characterized in that: include: A mode determination module, used to obtain a target AR application mode selected by a user from multiple AR application modes through a mode setting interface in an augmented reality AR container; a container processing module, configured to push the video stream data and 3D matrix data provided by the AR camera in the AR container to a 3D engine, or to push the video stream data and 3D matrix data to the 3D engine and a native AR engine in the AR container, according to the target AR application mode, to obtain AR content and 3D content, wherein the 3D engine includes a native 3D engine in the AR container or a webpage 3D engine outside the AR container; A view display module is used to construct at least two layers of views according to the target AR application mode, render the AR content and the 3D content on at least part of the views in the at least two layers of views, and obtain a comprehensive view of the AR application of the target AR application mode, wherein the at least two layers of views corresponding to different AR application modes have different structures and the views in which the AR content and the 3D content are rendered are different.

12. A device for developing and running augmented reality applications, characterized in that: include: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the method for developing and running an augmented reality application as described in any one of claims 1 to 10 is implemented.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the method for developing and running an augmented reality application according to any one of claims 1 to 10 is implemented.

14. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the method for developing and operating an augmented reality application as claimed in any one of claims 1 to 10.

Images