Intelligent augmented reality scene generation method and device based on three-dimensional large model

Abstract

The invention provides an intelligent augmented reality scene generation method and device based on a three-dimensional large model, and belongs to the technical field of augmented reality. The method comprises the following steps: capturing a current reality scene by using an augmented reality application program, and preprocessing captured image data; detecting a to-be-enhanced target object in the real scene image, and tracking the posture of the object; obtaining a virtual object generation instruction input by a user through the augmented reality application program; inputting an instruction into the three-dimensional generation large model to generate a virtual model required by a user; and placing the virtual model at an accurate position of a real scene by using an augmented reality rendering module, completing augmented reality virtual-real fusion, and displaying the virtual model to a user in an augmented reality application program. According to the method, the manufacturing cost of the augmented reality virtual scene is reduced, and the requirement of a user for a personalized augmented reality scene is met.

Description

Technical Field

[0001] The present invention belongs to the field of augmented reality technology, and specifically relates to a method and device for generating an intelligent augmented reality scene based on a three-dimensional large model. Background Art

[0002] Augmented reality technology uses computer technology to integrate virtual elements into the real environment to enhance the user's perception of the world. As people's demand for immersive and rich sensory experience grows, augmented reality is gradually playing an important role in various fields. However, on the one hand, the existing augmented reality methods can only superimpose pre-designed virtual models after identifying the real scene, which cannot meet the user's demand for specific scene experience. On the other hand, due to the limitations of 3D modeling technology, the process of generating virtual models is relatively complicated, and there is still a certain lack of realism and complexity.

[0003] Recently, AI big models have shown strong learning and content generation capabilities, which have the potential to solve complex problems in the real world. Among them, the emergence of 3D big models solves the problem of high-quality 3D content generation, mainly using diffusion model-based technology, which can generate 3D virtual models based on input text or 2D image prompts.

[0004] The 3D large model, with its powerful 3D modeling capabilities, reduces the cost of building virtual scenes and provides a convenient way for augmented reality users to create personalized experience needs. Combining the 3D large model with augmented reality technology can simplify the production process of AR virtual scenes, and users can get a more immersive augmented reality virtual-real fusion experience that meets their needs. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides an intelligent augmented reality scene generation method based on a three-dimensional large model, which is used to solve the problem that the personalized needs of users for augmented reality cannot be met and the generated virtual scene is subject to technical limitations.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0007] A method for generating an intelligent augmented reality scene based on a three-dimensional large model, comprising:

[0008] Using augmented reality applications to call the RGB camera through the user's mobile communication terminal to obtain the current real scene and perform data preprocessing;

[0009] The preprocessed results are identified using the target detection algorithm, and compared and matched with the target object data in the repository to obtain the name of the target object;

[0010] The pose of the target object is estimated through the 6D pose estimation model to obtain the pose information;

[0011] The user inputs a text description instruction or a 2D image that needs to be superimposed on the three-dimensional model of the target object through the augmented reality application;

[0012] Input the text description instruction or 2D image into the three-dimensional generation large model to generate a corresponding three-dimensional virtual model;

[0013] According to the posture information of the target object, the generated three-dimensional virtual model is superimposed on the position of the target object based on the distance of the position by using LOD technology, and is displayed to the user in the augmented reality application.

[0014] Furthermore, the preprocessed results are identified using a target detection algorithm and compared and matched with the target object data in the repository, including inputting the preprocessed screen image into a pre-trained target recognition model in the cloud. When the target object is detected, the category name of the target object is identified, the target object is selected with a box, and the augmented reality application is used to display it to the user through the mobile terminal.

[0015] Furthermore, the target detection algorithm includes a DETR algorithm, and the DETR algorithm is deployed in the cloud.

[0016] Furthermore, the pose estimation of the target object through the 6D pose estimation model includes extracting an RGBD image of the target object from a cloud repository based on the identified target object name, inputting the RGBD image into the 6D pose estimation model, and tracking the pose of the target object.

[0017] Furthermore, the use of LOD technology to superimpose the generated three-dimensional virtual model to the position of the target object based on the distance of the position includes setting a distance threshold. According to the LOD criterion, when the distance of the target object from the camera is less than the distance threshold, a high-quality model is rendered, and when the distance is greater than the distance threshold, a low-detail model is rendered, and superimposed rendering of virtual-real fusion is performed, which is displayed to the user on the terminal through an augmented reality application.

[0018] On the other hand, the present invention provides an intelligent augmented reality scene generation device based on a three-dimensional large model, which is implemented based on the aforementioned intelligent augmented reality scene generation method based on a three-dimensional large model, and is characterized in that it includes a scene capture module, a target tracking and recognition module, a three-dimensional virtual model generation module, and an augmented reality rendering module;

[0019] The scene capture module is used to use the augmented reality application to call the RGB camera through the user's mobile communication terminal to obtain the current real scene and perform data preprocessing, and transmit the preprocessed image to the target tracking and recognition module;

[0020] The target tracking and recognition module is used to identify the preprocessed results using the target detection algorithm, compare and match them with the target object data in the repository, and obtain the name of the target object; estimate the position and posture of the target object through the 6D posture estimation model to obtain the posture information, and input the posture information into the augmented reality rendering module;

[0021] A 3D virtual model generation module, based on a user's text description instruction or 2D image of the 3D model of the target object, generates a corresponding 3D virtual model using a 3D generation macro model and inputs it into an augmented reality rendering module;

[0022] The augmented reality rendering module uses LOD technology to overlay the generated three-dimensional virtual model on the position of the target object based on the posture information of the target object and the distance of the position, and displays it to the user in the augmented reality application.

[0023] In a third aspect, the present invention also provides an electronic device comprising: a processor and a storage device, wherein the storage device stores one or more programs that can be called by the processor; the one or more processors execute the aforementioned intelligent augmented reality scene generation method based on a three-dimensional large model by calling the computer program stored in the storage device.

[0024] In a fourth aspect, the present invention further provides a computer-readable storage medium having erasable executable instructions stored thereon; when the instructions are executed on a computer device, the computer device executes the aforementioned three-dimensional model-based intelligent augmented reality generation method in the background.

[0025] The beneficial effects of the present invention are:

[0026] In the past, after identifying the object to be enhanced, the augmented reality system could only overlay specific virtual elements pre-designed by the developer in the repository. The above-mentioned intelligent augmented reality scene generation method based on the three-dimensional large model provided by the present invention, by utilizing the powerful 3D modeling capability of the three-dimensional large model, can enable the user to generate various virtual objects required by the user through command interaction and overlay them into the real scene, thereby providing the user with a more realistic and immersive augmented reality experience, and solving the problem that traditional augmented reality is limited by traditional algorithm technology, lacks authenticity, and cannot meet the personalized customization needs of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of the method for generating an intelligent augmented reality scene based on a three-dimensional large model of the present invention;

[0028] Figure 2 It is a connection diagram of the intelligent augmented reality scene generation device based on the three-dimensional large model of the present invention;

[0029] Figure 3 It is a logic flow chart of the intelligent augmented reality scene generation method based on the three-dimensional large model of the present invention;

[0030] Figure 4 It is a schematic diagram of the structure of an electronic device of the present invention. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0032] See also Figure 1 The present invention provides a method for generating an intelligent augmented reality scene based on a three-dimensional large model, and the method includes steps S110~S160.

[0033] In operation S110, an augmented reality application is used to call an RGB camera through a user's mobile communication terminal to obtain a current real scene and perform data preprocessing.

[0034] The user opens the augmented reality application, uses the camera of the mobile communication terminal to obtain the current picture in the real world, and extracts key frames from the obtained picture image data for preprocessing.

[0035] In operation S120, the preprocessed result is identified and compared and matched with the target object data in the storage library.

[0036] First, the screen image preprocessed by operation S110 is input into the target recognition model pre-trained in the cloud. When the target object is detected, the category name of the object is identified, the object is selected with a box, and displayed to the user through the mobile terminal using an augmented reality application.

[0037] According to an embodiment of the present invention, the above-mentioned target recognition algorithm can use the DETR algorithm to identify and detect the target object. The DETR (DEtection TRansformer) algorithm uses the Transformer architecture to build a model and is an end-to-end target detection algorithm. The present invention deploys the DETR algorithm in the cloud and uses the DETR algorithm to detect the position of the target object to be enhanced. First, the input real-time image is feature extracted by the CNN convolutional neural network to obtain a feature map, and then the feature map is position-encoded and position information is added. Then, the global feature is constructed and feature interaction is performed using the Transformer's multi-head attention mechanism, and finally the position and name of the target are obtained through the output layer.

[0038] In operation S130 , a pose estimation is performed on the target object using a 6D pose estimation model.

[0039] First, according to the name of the target object identified in operation S120, an RGBD image of the target object is extracted from the cloud storage library, and then the RGBD image is input into the 6D posture estimation model to track the posture of the target object.

[0040] According to an embodiment of the present invention, the above-mentioned 6D posture estimation model can be a pre-trained FoundationPose. FoundationPose combines a large language model (LLM), a new transformer-based architecture, and a contrastive learning method to perform large-scale synthetic data training, and has strong generalization capabilities. Therefore, when estimating the posture of a new object, no fine-tuning is required. The present invention deploys the FoundationPose model in the cloud, inputs the RGBD image of the object into the FoundationPose model, models the object using a neural implicit field representation, and then initializes the translation and rotation postures. Then, the initial posture is updated using a Transformer-based posture optimization network, and the generated multiple hypothetical postures are scored, and the most accurate posture is selected as the output result to obtain the 6D posture information of the enhanced object in the current real-time image.

[0041] The above output posture is a 6D posture, namely, a displacement of 3 degrees of freedom and a spatial rotation (Rotation) of 3 degrees of freedom. Augmented reality rendering is performed according to the conversion relationship between coordinate systems.

[0042] In operation S140 , the user inputs a text description instruction or a 2D image that needs to be superimposed on the three-dimensional model of the scene object through the augmented reality application.

[0043] The user inputs the prompt of the virtual scene to be generated through the augmented reality application of the mobile terminal. The user can input text instructions to describe the three-dimensional virtual model to be generated, or input a 2D image of the three-dimensional virtual model.

[0044] In operation S150, the instruction is input into the three-dimensional large model generation to generate a corresponding virtual model.

[0045] According to the instructions or images, the 3D virtual model required by the user is generated through the 3D large model Direct3D. Direct3D is built using 3D Diffusion Transformer (3D-DiT) and can generate high-quality 3D content. Extract key instructions C based on the input content t As conditional information, we first generate a three-dimensional potential representation through the diffusion model to obtain the three-dimensional potential features. , the latent features are decoded to generate a high-resolution 3D geometric model M.

[0046] ,

[0047] The 3D model is then processed with LOD (Level of details), and the high-quality model generated by Direct3D is simplified into a low-detail model using mesh simplification. These two levels of 3D models are stored in a cloud repository.

[0048] In operation S160 , the generated three-dimensional virtual model is rendered to a correct position in the real environment according to the posture information of the real target object, and is displayed to the user in an augmented reality application.

[0049] According to the posture information of the target object to be enhanced in the real world obtained in operation S130, the three-dimensional virtual model obtained in operation S150 is superimposed on the accurate position of the target object in reality, and a distance threshold is set. According to the LOD criterion, when the distance between the object to be enhanced in the real world and the camera is less than the threshold, a high-quality model is rendered; when the distance is far and greater than the threshold, a low-detail model is rendered, and virtual-real fusion rendering is performed and displayed to the user on the terminal through an augmented reality application.

[0050] Figure 2 It is a module connection diagram of an intelligent augmented reality scene generation device based on a three-dimensional large model according to an embodiment of the present invention.

[0051] like Figure 2 As shown, the intelligent augmented reality scene generation method based on the three-dimensional large model includes a scene capture module, a target tracking and recognition module, a three-dimensional virtual model generation module and an augmented reality rendering module:

[0052] The scene capture module is used to call the real world scene captured by the camera through the user's mobile communication terminal, pre-process the captured scene image, and transfer the pre-processed image to the target tracking and recognition module;

[0053] Among them, the target tracking and identification module is used to identify and track the obtained pre-processed real scene image. First, the target object in the scene is detected by using the target detection algorithm to obtain the name of the target object. According to the name, the preset RGBD image of the target object is extracted from the cloud storage library and input into the 6D posture estimation model, the position and posture of the target object are tracked, and the posture information is input into the augmented reality module.

[0054] Among them, the 3D virtual model generation module uses three-dimensional large model generation to make relevant virtual models based on the user's description text or image of the required model and inputs them into the augmented reality rendering module.

[0055] Among them, the augmented reality rendering module superimposes the virtual model on the accurate position of the real object according to the posture of the target object and the generated virtual model, thereby completing the augmented reality virtual-real fusion rendering.

[0056] Figure 3 It is a logic flow chart of a method for generating an intelligent augmented reality scene based on a three-dimensional large model according to an embodiment of the present invention.

[0057] like Figure 3 As shown, the user captures the image of the current real-world scene through a mobile terminal phone or other device with an RGB camera (tablet, AR glasses, etc.), and uses the augmented reality application embedded in the mobile terminal to preprocess the image, and then transmits it to the cloud for detection and recognition of the target object. If the target object to be enhanced is not detected in the picture, the real scene picture continues to be captured; if the target object to be enhanced is detected in the picture, the RGBD image of the object is extracted from the cloud repository according to the recognized object name, and the RGBD image is input into the 6D posture estimation model to track the object posture. At the same time, the user inputs the description text or 2D image of the 3D virtual model to be superimposed through the augmented reality application on the mobile terminal, and inputs the instruction into the 3D large model in the cloud to generate the corresponding virtual model. Finally, rendering is performed according to the posture information of the target object and the 3D virtual model to realize the virtual-real fusion of augmented reality.

[0058] Figure 4 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention.

[0059] like Figure 4As shown, the electronic device 400 may include one or more processors and one or more memories. The memories store computer-readable codes, which, when executed by one or more processors, may execute the above-mentioned method for generating an intelligent augmented reality scene based on a three-dimensional large model. Figure 4 The electronic device architecture shown in FIG. Figure 4 As shown, the electronic device 400 may include a bus 401, one or more processors 402, ROM 403, RAM 404, a communication port 405 connected to a network, an input / output component 406, a memory 407, etc. The storage device in the electronic device 400, such as ROM 403 or memory 407, may store the intelligent augmented reality scene generation method based on a three-dimensional large model provided by the present invention.

[0060] The present invention also provides a computer-readable storage medium, which stores computer-readable instructions, including but not limited to, for example, volatile memory and / or non-volatile memory. Volatile memory may include, for example, RAM.

[0061] and cache, etc. The non-volatile memory may include, for example, ROM, hard disk, flash memory, etc. The computer-readable storage medium carries one or more programs, and when the one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0062] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for generating an intelligent augmented reality scene based on a three-dimensional large model, characterized in that: include: Using augmented reality applications to call the RGB camera through the user's mobile communication terminal to obtain the current real scene and perform data preprocessing; The preprocessed results are identified using the target detection algorithm, and compared and matched with the target object data in the repository to obtain the name of the target object; The pose of the target object is estimated through the 6D pose estimation model to obtain the pose information; The user inputs a text description instruction or a 2D image that needs to be superimposed on the three-dimensional model of the target object through the augmented reality application; Input the text description instruction or 2D image into the three-dimensional generation large model to generate a corresponding three-dimensional virtual model; According to the posture information of the target object, the generated three-dimensional virtual model is superimposed on the position of the target object based on the distance of the position by using LOD technology, and is displayed to the user in the augmented reality application.

2. The method for generating an intelligent augmented reality scene based on a three-dimensional large model according to claim 1, characterized in that: The preprocessed results are identified using a target detection algorithm and compared and matched with the target object data in the repository, including inputting the preprocessed screen image into a target recognition model pre-trained in the cloud. When the target object is detected, the category name of the target object is identified, the target object is selected with a box, and the augmented reality application is used to display it to the user through the mobile terminal.

3. The method for generating an intelligent augmented reality scene based on a three-dimensional large model according to claim 2, characterized in that: The target detection algorithm includes a DETR algorithm, and the DETR algorithm is deployed in the cloud.

4. The method for generating an intelligent augmented reality scene based on a three-dimensional large model according to claim 1, characterized in that: The pose estimation of the target object by the 6D pose estimation model includes extracting an RGBD image of the target object from a cloud storage repository based on the identified target object name, inputting the RGBD image into the 6D pose estimation model, and tracking the pose of the target object.

5. The method for generating an intelligent augmented reality scene based on a three-dimensional large model according to claim 1, characterized in that: The method of using LOD technology to superimpose the generated three-dimensional virtual model to the position of the target object based on the distance of the position includes setting a distance threshold. According to the LOD criterion, when the distance of the target object from the camera is less than the distance threshold, a high-quality model is rendered, and when the distance is greater than the distance threshold, a low-detail model is rendered, and superimposed rendering of virtual-real fusion is performed, which is displayed to the user on the terminal through an augmented reality application.

6. An intelligent augmented reality scene generation device based on a three-dimensional large model, which is implemented based on the intelligent augmented reality scene generation method based on a three-dimensional large model according to any one of claims 1 to 5, characterized in that: It includes scene capture module, target tracking and recognition module, 3D virtual model generation module and augmented reality rendering module; The scene capture module is used to use the augmented reality application to call the RGB camera through the user's mobile communication terminal to obtain the current real scene and perform data preprocessing, and transmit the preprocessed image to the target tracking and recognition module; The target tracking and identification module is used to identify the preprocessed results using the target detection algorithm, compare and match them with the target object data in the repository, and obtain the name of the target object; The pose of the target object is estimated through the 6D pose estimation model to obtain the pose information, and the pose information is input into the augmented reality rendering module; A 3D virtual model generation module, based on a user's text description instruction or 2D image of the 3D model of the target object, generates a corresponding 3D virtual model using a 3D generation macro model and inputs it into an augmented reality rendering module; The augmented reality rendering module uses LOD technology to overlay the generated three-dimensional virtual model on the position of the target object based on the posture information of the target object and the distance of the position, and displays it to the user in the augmented reality application.

7. An electronic device, characterized in that: include: processor and storage device; Wherein, the storage device stores one or more programs that can be called by the processor; The one or more processors execute the intelligent augmented reality scene generation method based on a three-dimensional large model as described in any one of claims 1-5 by calling the computer program stored in the storage device.

8. A computer-readable storage medium, characterized in that: Erasable executable instructions are stored thereon; when the instructions are executed on a computer device, the computer device executes the intelligent augmented reality generation method based on a three-dimensional model as described in any one of claims 1-5 in the background.

Images