Platform WebAR implementation method and system, storage medium and electronic equipment

Abstract

The invention belongs to the technical field of augmented reality, and provides a platform WebAR implementation method and system, a storage medium and electronic equipment, and the method comprises the steps: collecting environment information, and building a 3D model and a WebAR management system; the WebAR management system comprises a front end and a rear end; obtaining and processing a video stream based on a WebAR management system, and determining an AR material; and the front end performs rendering based on the AR material and realizes real-time tracking and positioning of the 3D model. According to the invention, high expansibility is supported by adopting a technical architecture with separated front and rear ends, and platform experience is provided by configuring the environment information identifier and the AR material content.

Description

Technical Field

[0001] The present invention belongs to the field of augmented reality technology, and in particular relates to a platform-based WebAR implementation method, system, storage medium and electronic device. Background Art

[0002] AR, Augmented Reality, is a technology that cleverly integrates virtual information with the real world. AR uses a variety of technical means such as 3D modeling, multimedia, real-time tracking and intelligent interaction to simulate virtual information such as text, images, 3D models, music, and videos, and then loads them into the real world. Therefore, virtual information and real-world information complement each other to achieve "enhancement" of the real world.

[0003] At present, the mainstream AR application development in the market mainly relies on the following devices: head-mounted AR devices (HMD devices, such as RokidAIR), smart glasses (such as Google Glass Enterprise Edition 2) and handheld devices (such as smartphones, tablets, etc.). However, this type of AR solution has certain drawbacks, which are manifested in:

[0004] (1) Many AR solutions require users to download and install dedicated native applications, which not only takes up storage space on the user's device, but also increases development costs because native AR applications need to be developed and maintained for different operating systems (such as iOS and Android) and may also cause compatibility issues between different platforms.

[0005] (2) At present, device-based AR application development is often limited by device performance, which affects the quality and fluency of AR content. When running AR applications on low-performance devices, users may experience poor experiences such as freezes and delays. In addition, the high cost of high-performance AR devices also limits their popularity and application scope.

[0006] (3) In terms of native content update management, users usually need to re-download or update the application to obtain the latest content. This process increases the complexity of maintenance and the user's operation steps. Summary of the invention

[0007] To solve the above problems, the present invention provides a platform-based WebAR implementation method, system, storage medium and electronic device.

[0008] A platform-based WebAR implementation method of the present invention comprises:

[0009] Collect environmental information and establish a 3D model and a WebAR management system; the WebAR management system includes a front end and a back end;

[0010] Acquire and process video streams based on the WebAR management system to determine AR materials;

[0011] The front end performs rendering based on the AR material and realizes real-time tracking and positioning of the 3D model.

[0012] Furthermore, the collecting of environmental information and establishing of 3D model and WebAR management system specifically includes:

[0013] Collecting environmental information based on the selected scene, the environmental information including pictures and audio, the pictures serving as scene identification files, and the audio serving as scene audio files;

[0014] Based on the scene identification file, a 3D model is established using a modeling tool, and a scene model file is exported;

[0015] Establishing an environment feature vector based on the environment information and the scene model file;

[0016] A WebAR management system is established to store the environmental feature vector.

[0017] Furthermore, the environmental feature vector includes: scene unique ID, scene name, scene introduction, scene identification object, scene audio object, scene model object, scene model introduction, model scaling ratio, feature point similarity, and minimum number of feature points.

[0018] Furthermore, the WebAR management system adopts a development method in which the front-end and the back-end are separated, and the back-end communicates with the front-end via the https protocol.

[0019] Furthermore, the WebAR management system acquires and processes the video stream and determines the AR material, specifically including:

[0020] The front end calls the camera to obtain the video stream, extracts a frame of image from the video stream based on the scheduled time, converts it into Base64 encoding, and transmits it to the back end through the https protocol;

[0021] The backend uses OpenCV technology to match the feature points of the video stream frame image with the scene identification file and determine the key coordinates of the feature points;

[0022] According to the key coordinates of the feature points, a homography matrix is ​​determined to calculate the position and rotation angle of the 3D model at the front end;

[0023] According to the matching results of the video stream frame image and the scene identification file, the scene unique ID, scene name, scene introduction, scene identification object, scene audio object, model scaling, model position and rotation angle of the environment feature vector are returned to the front end;

[0024] The front end calls the back end download interface through the https protocol and downloads the scene model file and scene audio file.

[0025] Furthermore, the backend uses OpenCV technology to match feature points between the video stream frame image and the scene identification file to determine the key coordinates of the feature points, specifically including:

[0026] Extract feature points of video stream frame images and scene identification files;

[0027] The video stream frame image is matched with the feature points of the scene identification file, and the similarity of the feature points and the minimum number of feature points are combined to determine whether the video stream frame image matches the scene identification file and determine the key coordinates of the feature points.

[0028] Furthermore, the front end performs rendering based on the AR material and realizes real-time tracking and positioning of the 3D model, specifically including:

[0029] Use 3D rendering tools to render the scene model file, scene audio file, and the corresponding parameters of the environment feature vector;

[0030] Set the 3D model's rendering tag to manual control;

[0031] The front end compares the video stream frame image with the scene identification file based on the predetermined time, re-obtains the return value of the 3D model, model scaling, model position and rotation angle, and updates it to the 3D rendering tag.

[0032] The present invention also provides a platform-based WebAR implementation system, characterized in that the system comprises:

[0033] A collection and construction module, used to collect environmental information and build a 3D model and a WebAR management system; the WebAR management system includes a front end and a back end;

[0034] AR material determination module, used to obtain and process video streams based on the WebAR management system to determine AR materials;

[0035] The rendering and positioning module is used for front-end rendering based on AR materials and to achieve real-time tracking and positioning of 3D models.

[0036] The present invention also proposes a computer-readable storage medium storing a program or instruction. When the program or instruction is executed on a computer, the computer executes the aforementioned platform-based WebAR implementation method.

[0037] The present invention also proposes an electronic device, including a processor, which is coupled to a memory; the processor is used to read and execute a computer program stored in the memory to implement the aforementioned platform-based WebAR implementation method.

[0038] Compared with the prior art, the present invention has the following advantages:

[0039] 1. Platformization and flexibility. Users can manage and configure AR content in different environments through one platform, thereby improving content management efficiency and consistency. At the same time, it supports users to configure AR experience according to different environments to achieve a more personalized experience. In the specific use process, the back-end system supports the functions of content upload, download and management, ensuring that the update and maintenance of AR content are more efficient, thereby improving the use effect.

[0040] 2. Simplify user operations and avoid device performance drawbacks. Without installing additional applications, users can directly access the AR scanning page through the browser, quickly load and experience related AR content, and lower the user's entry threshold. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 A schematic diagram of a flow chart of a platform-based WebAR implementation method according to an embodiment of the present invention;

[0043] Figure 2 It is an ER relationship diagram of the database structure in the embodiment of the present invention;

[0044] Figure 3 A schematic diagram of the front-end and back-end technical architecture of a Web system according to an embodiment of the present invention;

[0045] Figure 4 A schematic diagram of a Web system architecture according to an embodiment of the present invention;

[0046] Figure 5 A schematic diagram of the structure of a platform-based WebAR implementation system according to an embodiment of the present invention;

[0047] Figure 6 It is a schematic structural diagram of the electronic device of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] In addition, in order to have a deeper understanding of the present invention, some of the professional terms are explained as follows:

[0050] A-Frame is a powerful and easy-to-use tool for rapid development of WebVR and WebAR applications. It lowers the threshold for creating virtual reality and augmented reality content by simplifying the development process, providing cross-platform support and rich community resources.

[0051] Blender is a powerful and flexible 3D creation tool suitable for a variety of professional and personal projects. Its open source and free features, comprehensive functions and wide range of applications make it an important tool in the 3D creation industry.

[0052] OpenCV (Open Source Computer Vision Library) is a cross-platform computer vision library released under the BSD license. It can be used to develop real-time image processing, computer vision, and pattern recognition programs.

[0053] In one embodiment of the present invention, a platform-based WebAR implementation method is provided, such as Figure 1 As shown, the method includes:

[0054] S1. Collect environmental information and establish a 3D model and WebAR management system. The WebAR management system includes a front end and a back end, specifically including:

[0055] S11. Collect environmental information based on the selected scene, the environmental information including pictures and audio, the pictures serving as scene identification files, and the audio serving as scene audio files; wherein the collection method is not limited, and the picture format is png or jpg.

[0056] S12. Based on the scene identification file, use a modeling tool to build a 3D model and export a scene model file; wherein the format of the scene model file is gltf; the modeling tool may be Blender or C4D (Cinema 4D).

[0057] S13, establishing an environment feature vector based on the environment information and the scene model file, wherein the environment feature vector includes: scene unique ID, scene name, scene introduction, scene identification object, scene audio object, scene model object, scene model introduction, model scaling ratio, feature point similarity, and minimum number of feature points;

[0058] Among them, scene identification objects, scene audio objects and scene model objects are stored in object mode; model scaling and feature point similarity are stored in floating point mode; the minimum number of feature points is stored in integer mode;

[0059] The scene identification file, scene audio file and scene model file are stored in the object storage database, and the object description information is stored in the relational database table. The structure of the relational database table is: file unique ID, object storage ID, file name and file description, such as Figure 2 As shown;

[0060] S14, establish a WebAR management system to store the environmental feature vector, the WebAR management system adopts a front-end and back-end separation development method, such as Figure 3 and 4 As shown;

[0061] The front end includes upload, download, display and modification pages, camera shooting, model rendering and positioning modules;

[0062] The backend includes upload, download, modification, environment feature comparison interfaces and storage modules;

[0063] The backend communicates with the frontend via the https protocol.

[0064] S2. Obtain and process the video stream based on the WebAR management system and determine the AR material, including:

[0065] S21. The front end calls the camera to obtain the video stream, extracts a frame of image from the video stream every 100 milliseconds, converts it into Base64 encoding, and transmits it to the back end through the https protocol; among them, the A-Frame framework and WebRTC technology (Web Real-Time Communication web page video and voice real-time communication technology) can be used to call the camera and obtain the video stream in real time.

[0066] S22. The backend uses OpenCV technology to match feature points between the video stream frame image and the scene identification file to determine the key coordinates of the feature points, including:

[0067] Extract feature points of video stream frame images and scene identification files; wherein the feature point detection algorithm can be the ORB algorithm of the OpenCV library;

[0068] Match the feature points of the video stream frame image with the scene identification file, and determine whether the video stream frame image matches the scene identification file by combining the feature point similarity and the minimum number of feature points, and determine the key coordinates of the feature points; wherein the feature point matching algorithm can be the BFMatcher algorithm in the OpenCV library;

[0069] S23, determining the homography matrix according to the key coordinates of the feature points, and calculating the position and rotation angle of the 3D model at the front end;

[0070] S24, according to the matching result of the video stream frame image and the scene identification file, the scene unique ID, scene name, scene introduction, scene identification object, scene audio object, model scaling ratio, model position and rotation angle of the environment feature vector are returned to the front end;

[0071] The front end calls the back end download interface through the https protocol and downloads the scene model file and scene audio file.

[0072] S3. The front end performs rendering based on the AR material and realizes real-time tracking and positioning of the 3D model, specifically including:

[0073] S31. Use A-Frame's 3D rendering engine to render the scene model file, scene audio file, and the corresponding parameters of the environment feature vector. The scene model file is rendered using<a-entity gltf-model / > Tags, scene audio file usage <a-sound>Label, environment feature vector corresponding parameters use<a-entity text> Label;

[0074] The parameters for rendering the 3D model are the model scaling, model position and rotation angle returned by the environment feature vector;

[0075] S32, render the 3D model <a-entity>The matrixAutoUpdate property is set to false for manual control, which means manual control of the model rendering on the front end;

[0076] Every 100 milliseconds, the front end compares the video stream frame image with the scene identification file, re-obtains the return value of the 3D model, model scale, model position and rotation angle, and uses the DOM API to update the 3D rendering tag in the A-Frame framework <a-entity>superior.

[0077] The embodiment of the present invention also provides a platform-based WebAR implementation system, such as Figure 5 As shown, the system comprises:

[0078] The acquisition and construction module 501 is used to acquire environmental information and establish a 3D model and a WebAR management system; the WebAR management system includes a front end and a back end;

[0079] The AR material determination module 502 is used to obtain and process the video stream based on the WebAR management system to determine the AR material;

[0080] The rendering and positioning module 503 is used for the front-end to render based on AR materials and realize real-time tracking and positioning of the 3D model.

[0081] An embodiment of the present invention further provides a computer-readable storage medium storing a program or instruction. When the program or instruction is executed on a computer, the computer executes a platform-based WebAR implementation method as described in the above method embodiment.

[0082] like Figure 6 As shown, an embodiment of the present invention further provides an electronic device, including: a processor 601, the processor 601 is coupled to a memory 602, and the processor 601 is used to read and execute a computer program stored in the memory 602 to implement a platform-based WebAR implementation method as described in the above method embodiment.

[0083] Embodiment 1

[0084] S1. Use a professional camera to shoot the selected scene to obtain environmental information. The picture is used as a scene identification file and is named scene.png. Use audio recording software to introduce the selected scene. The audio is used as a scene audio file and is named vedio.mp3.

[0085] Based on the scene identification file, use the modeling tool C4D to create a 3D model and export the scene model file scene.gltf.

[0086] Based on the environment information and the scene model file, an environment feature vector is established. The environment feature vector includes: scene unique ID, scene name, scene introduction, scene identification object, scene audio object, scene model object, scene model introduction, model scaling ratio, feature point similarity, and minimum number of feature points;

[0087] Among them, the scene unique ID is generated using the ObjectId provided by the MongoDB component;

[0088] In this embodiment, the scene unique ID [5ab9cbfa31c2ab715d42129e], scene name [first floor gate], scene introduction [welcome to all technical representatives], scene identification object [file unique ID corresponding to scene.png], scene audio object [file unique ID corresponding to vedio.mp3], scene model object [file unique ID corresponding to scene.gltf], scene model introduction [bright and glorious life], model scaling ratio [0.5], feature point similarity [0.7], minimum number of feature points

[27] .

[0089] Establish a WebAR management system and use front-end development technology [vue / html] to build the front-end page. The front-end page contains input forms for scene name, scene introduction, scene model introduction, model scaling, feature point similarity and minimum number of feature points, as well as upload attachment forms for scene identification files, scene audio files and scene model files;

[0090] The front end uploads the form data to the upload interface of the back end through the https protocol; the upload interface stores the form data in the object storage database Minio, and stores the object description information of the form data in the resource table of oracle; the resource table fields include: file unique ID, object storage ID, file name and file description.

[0091] The front end uses the a-videosphere component of A-Frame to call the camera. During the call, the getUserMedia method is used to authorize the user's camera permission to obtain the video stream, extract a frame of image from the video stream every 100 milliseconds, convert it into Base64 encoding, and transmit it to the back end through the https protocol.

[0092] Use Python's ORM framework SQLAlchemy to obtain video stream frames and scene identification files, and use the ORB algorithm of the OpenCV library to extract feature points of video stream frames and scene identification files. The feature points are "image descriptors" calculated by the ORB algorithm.

[0093] Use the BFMatcher algorithm of the OpenCV library to determine whether the video stream frame image matches the scene identification file and determine the key coordinates of the feature points based on the feature point "image descriptor", feature point similarity and the minimum number of feature points.

[0094] According to the key coordinates of the feature points, the findHomography method provided by OpenCV is used to estimate the homography matrix, and the position and rotation angle of the 3D model at the front end are calculated through the PnP solution function solvePnP provided by OpenCV. The calculation process is as follows:

[0095] Extract the coordinates of the key points of the Blender model: extract the three-dimensional world coordinates of the key points from the Blender model to obtain the coordinate set A;

[0096] Apply the homography matrix to perform coordinate transformation: Use the homography matrix to calculate the projection set of the 3D model on 2D through the perspectiveTransform function of OpenCV to obtain the projection set B;

[0097] Get the camera intrinsic matrix: Use A-Frame's getObject3D method to get the camera object, and combine it with Three.js's Matrix4 method to get the camera's intrinsic matrix C;

[0098] Estimate the position and orientation of the 3D model in the 2D scene: Use OpenCV.solvePnP(set A, set B, matrix C) to estimate the rotation vector rvec and translation vector tvec of the 3D model mapped to the 2D scene identification file, thereby determining the position and rotation angle of the 3D model at the front end.

[0099] According to the matching results of the video stream frame image and the scene identification file, the scene unique ID [5ab9cbfa31c2ab715d42129e], scene name [first floor gate], scene introduction [welcome to all technical representatives], scene identification object [file unique ID corresponding to scene.png], scene audio object [file unique ID corresponding to vedio.mp3], model scaling ratio [0.5], model position and rotation angle of the environment feature vector are returned to the front end;

[0100] The front end calls the backend download interface through the https protocol based on the file unique ID of the scene identification object and the file unique ID of the scene audio object. The download interface uses the python minio client to download the scene model file and scene audio file.

[0101] A-Frame's 3D rendering engine Three.js is used to render the scene model file, scene audio file, and the corresponding parameters of the environmental feature vector. The scene model file is rendered using<a-entity gltf-model / > Tags, scene audio file usage <a-sound>Label, environment feature vector corresponding parameters use<a-entity text> Label;

[0102] The parameters for rendering the 3D model are the model scale, model position and rotation angle returned by the environment feature vector;

[0103] Rendering tags for 3D models <a-entity>Set the matrixAutoUpdate property to false to control it manually;

[0104] Every 100 milliseconds, the front end compares the video stream frame with the scene identification file, re-obtains the return value of the 3D model, model scale, model position and rotation angle, and sets it to the docment.getElementById <a-entity>The object3D.matrix property of the tag is used to realize real-time tracking and positioning of the 3D model.

[0105] The open source tools used in the implementation process of the present invention, such as the modeling tool blender, can also be replaced by other modeling tools such as 3Dmax; the corresponding A-frame front-end implementation technology can also be replaced by other open source frameworks such as webXR, WebGL, etc.

[0106] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent substitutions for some of the technical features therein; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A platform-based WebAR implementation method, characterized in that: The method comprises: Collect environmental information and establish a 3D model and a WebAR management system; the WebAR management system includes a front end and a back end; Acquire and process video streams based on the WebAR management system to determine AR materials; The front end performs rendering based on the AR material and realizes real-time tracking and positioning of the 3D model.

2. The method according to claim 1, characterized in that: The collecting of environmental information and establishing of 3D model and WebAR management system specifically includes: Collecting environmental information based on the selected scene, the environmental information including pictures and audio, the pictures serving as scene identification files, and the audio serving as scene audio files; Based on the scene identification file, a 3D model is established using a modeling tool, and a scene model file is exported; Establishing an environment feature vector based on the environment information and the scene model file; A WebAR management system is established to store the environmental feature vector.

3. The method according to claim 2, characterized in that The environmental feature vector includes: scene unique ID, scene name, scene introduction, scene identification object, scene audio object, scene model object, scene model introduction, model scaling ratio, feature point similarity, and minimum number of feature points.

4. The method according to claim 2, characterized in that: The WebAR management system adopts a development method in which the front-end and the back-end are separated, and the back-end communicates with the front-end via the https protocol.

5. The method according to claim 1, characterized in that The WebAR management system acquires and processes the video stream and determines the AR material, specifically including: The front end calls the camera to obtain the video stream, extracts a frame of image from the video stream based on the scheduled time, converts it into Base64 encoding, and transmits it to the back end through the https protocol; The backend uses OpenCV technology to match the feature points of the video stream frame image with the scene identification file and determine the key coordinates of the feature points; According to the key coordinates of the feature points, a homography matrix is ​​determined to calculate the position and rotation angle of the 3D model at the front end; According to the matching results of the video stream frame image and the scene identification file, the scene unique ID, scene name, scene introduction, scene identification object, scene audio object, model scaling, model position and rotation angle of the environment feature vector are returned to the front end; The front end calls the back end download interface through the https protocol and downloads the scene model file and scene audio file.

6. The method according to claim 5, characterized in that The backend uses OpenCV technology to match feature points between the video stream frame image and the scene identification file to determine the key coordinates of the feature points, specifically including: Extract feature points of video stream frames and scene identification files; The video stream frame image is matched with the feature points of the scene identification file, and the similarity of the feature points and the minimum number of feature points are combined to determine whether the video stream frame image matches the scene identification file and determine the key coordinates of the feature points.

7. The method according to claim 3, characterized in that The front end performs rendering based on the AR material and realizes real-time tracking and positioning of the 3D model, specifically including: Use 3D rendering tools to render the scene model file, scene audio file, and the corresponding parameters of the environment feature vector; Set the 3D model's rendering tag to manual control; The front end compares the video stream frame image with the scene identification file based on the predetermined time, re-obtains the return value of the 3D model, model scaling, model position and rotation angle, and updates it to the 3D rendering tag.

8. A platform-based WebAR implementation system, characterized in that: The system comprises: A collection and construction module, used to collect environmental information and build a 3D model and a WebAR management system; the WebAR management system includes a front end and a back end; AR material determination module, used to obtain and process video streams based on the WebAR management system to determine AR materials; The rendering and positioning module is used for front-end rendering based on AR materials and to achieve real-time tracking and positioning of 3D models.

9. A computer-readable storage medium, characterized in that: A program or instruction is stored, and when the program or instruction is executed on a computer, the computer executes a platform-based WebAR implementation method as described in any one of claims 1 to 7.

10. An electronic device, characterized in that: comprising a processor coupled to a memory; The processor is used to read and execute the computer program stored in the memory to implement a platform-based WebAR implementation method as described in any one of claims 1 to 7.

Images