An augmented reality (AR) glasses and a display method, device and storage medium thereof

Abstract

The application provides an augmented reality (AR) glasses, a display method and device thereof and a storage medium. The method comprises the following steps: collecting a real scene image of an external environment by using a camera, and sending the collected real scene image to an image processing unit; performing three-dimensional reconstruction on the real scene image by using the image processing unit to obtain a three-dimensional image, and sending the three-dimensional image to an optical waveguide structure; and performing re-projection and image stabilization display on the three-dimensional image by using the optical waveguide structure, so as to display the three-dimensional image on a display lens of the AR glasses. The AR glasses provided by the application can realize what you see is what you get in a shooting scene.

Description

Technical Field

[0001] This application relates to the field of AR glasses display technology, and in particular to an augmented reality (AR) glasses display method, device and storage medium thereof. Background Technology

[0002] AR glasses are devices that link the virtual and real worlds, allowing users to see both real-world and virtual content.

[0003] In related technologies, AR glasses are usually equipped with a camera for capturing the external scene. After the camera captures the external scene, the optical projection component of the AR glasses will project the captured external scene onto the lenses of the AR glasses for the user to view.

[0004] However, in related technologies, the camera is generally placed on the side of the lens, which means there is a certain angle deviation between the camera and the lens. This causes a certain angle deviation between the scene captured by the camera and the scene that the user wants to see through the AR glasses. In this case, if the scene captured by the camera is directly projected onto the lens of the AR glasses, the angle of the scene presented on the lens will not match the user's viewing angle, affecting the user experience. Summary of the Invention

[0005] This application provides augmented reality (AR) glasses, a display method, a device, and a storage medium thereof, to at least solve the technical problem in the related art where errors in the camera results lead to a reduction in the realism of the feedback scene.

[0006] A first aspect of this application provides a method for displaying augmented reality (AR) glasses, the AR glasses including a camera, an image processing device, and an optical waveguide structure, the method comprising:

[0007] The system uses a camera to capture images of real-world scenes and then sends these images to an image processing unit.

[0008] The image processing unit performs three-dimensional reconstruction on the real scene image to obtain a three-dimensional image, and then sends the three-dimensional image to the optical waveguide structure;

[0009] The three-dimensional image is reprojected and stabilized using the optical waveguide structure to display the three-dimensional image on the display lens of the AR glasses.

[0010] A second aspect of this application provides an AR glasses display device, the AR glasses including a camera, an image processing device, and an optical waveguide structure, the device comprising:

[0011] The camera is used to capture images of real-world scenes and send them to the image processing unit.

[0012] An image processing unit is used to perform three-dimensional reconstruction on the real scene image to obtain a three-dimensional image, and send the three-dimensional image to the optical waveguide structure;

[0013] An optical waveguide structure is used to reproject and stabilize the three-dimensional image for display on the display lens of the AR glasses.

[0014] A third aspect of this application provides an AR glasses system, which includes a camera, an image processing device, and an optical waveguide structure. The AR glasses are configured to run computer instructions stored in a computer-readable storage medium. After the AR glasses executable instructions are executed by a processor, they can implement the method described in the first aspect above.

[0015] The fourth aspect of this application provides a computer-readable storage medium, wherein computer instructions are stored on the computer-readable storage medium; when the computer instructions are executed by a processor, they can implement the method described in the first aspect above.

[0016] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects:

[0017] In summary, the augmented reality (AR) glasses, display method, device, and storage medium proposed in this application first utilize a camera to capture images of a real-world scene and send these images to an image processing unit. Then, the image processing unit performs 3D reconstruction on the real-world scene images to obtain a 3D image, which is then sent to an optical waveguide structure. Finally, the optical waveguide structure is used to reproject and stabilize the 3D image, displaying it on the display lenses of the AR glasses. By reconstructing the real-world scene images to obtain a 3D image and then reprojecting and stabilizing the reconstructed 3D image onto the lenses, the user's viewing discomfort caused by the angle discrepancy between the camera and the AR glasses lenses can be compensated for, improving the realism of the displayed scene and enhancing the user experience, thus possessing significant application value.

[0018] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0019] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0020] Figure 1 This is a flowchart of an AR glasses display method according to an embodiment of this application;

[0021] Figure 2 This is a flowchart of an AR glasses display method according to an embodiment of this application;

[0022] Figure 3 This is a structural diagram of an AR glasses display device provided according to an embodiment of this application. Detailed Implementation

[0023] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0024] The augmented reality (AR) glasses and their display method proposed in this application first utilize a camera to capture images of a real-world scene and send these images to an image processing unit. Then, the image processing unit performs 3D reconstruction on the real-world scene images to obtain a 3D image, which is then sent to an optical waveguide structure. Finally, the optical waveguide structure is used to reproject and stabilize the 3D image, displaying it on the display lenses of the AR glasses. By reconstructing the real-world scene images to obtain a 3D image and then reprojecting and stabilizing the reconstructed 3D image onto the lenses, the user's viewing discomfort caused by the angle discrepancy between the camera and the AR glasses lenses can be compensated for, improving the realism of the displayed scene and enhancing the user experience, thus possessing significant application value.

[0025] The augmented reality (AR) glasses display method and AR glasses display device according to embodiments of this application are described below with reference to the accompanying drawings.

[0026] Example 1

[0027] Figure 1 A flowchart illustrating an AR (Augmented Reality) glasses display method provided in this application embodiment is shown below. Figure 1 As shown, the method may include:

[0028] Step 101: Use a camera to capture real-world scene images and send the captured images to the image processing unit.

[0029] In this embodiment of the disclosure, the camera may capture images of real-world scenes, for example, by taking videos or photographs.

[0030] Step 102: Use the image processing unit to perform three-dimensional reconstruction on the real scene image to obtain a three-dimensional image, and send the three-dimensional image to the optical waveguide structure.

[0031] For example, in this embodiment of the disclosure, the method for obtaining a 3D image by using the image processing unit to perform 3D reconstruction of the real scene image in this step may include:

[0032] Image information from a real-world scene is input into a 3D reconstruction model to obtain corresponding 3D mesh data. This 3D reconstruction model is pre-trained using a neural network algorithm. Then, the 3D mesh data is used to perform 3D reconstruction to obtain a 3D image.

[0033] In the embodiments of this disclosure, before performing step 102, the following step may be included: training the three-dimensional reconstruction model.

[0034] The methods for training a 3D reconstruction model can mainly include: first, determining test samples, such as real scene images from different angles captured by a camera in the past; then, training the 3D reconstruction model multiple times based on the test samples and calculating the convergence coefficient of the 3D reconstruction model; and confirming that the training is complete when the convergence coefficient meets the preset conditions.

[0035] It should be noted that the above training steps can be implemented in a computer device. After obtaining the trained 3D reconstruction model, the computer device sends the trained 3D reconstruction model to the AR glasses. At this time, the AR glasses need to send the images captured by its camera to the computer device so that the computer device can train the 3D reconstruction model. Alternatively, the above training steps can also be implemented in the AR glasses themselves.

[0036] Step 103: Use the optical waveguide structure to reproject and stabilize the three-dimensional image to display the three-dimensional image on the display lens of the AR glasses.

[0037] It should be noted that, in this embodiment, the optical waveguide structure can be used to perform optical projection imaging on an incoming light beam containing virtual image information, so that the virtual image information is displayed on the display lens of the AR glasses. Furthermore, in this embodiment, the optical waveguide structure can employ a reprojection-stabilized image display method during optical projection imaging, thus ensuring the final display effect.

[0038] Furthermore, in the embodiments of this application, both the camera and the optical waveguide structure are disposed on the side of the display lens; wherein, the camera and the optical waveguide structure are disposed on the same side of the display lens; or, the camera and the optical waveguide structure are disposed on opposite sides of the display lens.

[0039] In summary, the augmented reality (AR) glasses and display method proposed in this application first utilize a camera to capture images of a real-world scene and send these images to an image processing unit. Then, the image processing unit performs 3D reconstruction on the real-world scene images to obtain a 3D image, which is then sent to an optical waveguide structure. Finally, the optical waveguide structure is used to reproject and stabilize the 3D image, displaying it on the display lenses of the AR glasses. By reconstructing the real-world scene images to obtain a 3D image and then reprojecting and stabilizing the reconstructed 3D image onto the lenses, the user's viewing discomfort caused by the angle discrepancy between the camera and the AR glasses lenses can be compensated for, improving the realism of the displayed scene and enhancing the user experience, thus demonstrating significant application value.

[0040] Example 2

[0041] Figure 2 A flowchart of an AR glasses display method provided in an embodiment of this application is shown below. Figure 2 As shown, the method may include:

[0042] Step 201: Use a camera to capture real-world scene images and send the captured images to the image processing unit.

[0043] Step 202: Use the image processing unit to perform three-dimensional reconstruction on the real scene image to obtain a three-dimensional image, and send the three-dimensional image to the optical waveguide structure.

[0044] Step 203: Calculate the optimal projection angle using the image processing unit and send the optimal projection angle to the optical waveguide structure.

[0045] For example, the method for calculating the optimal projection angle may include:

[0046] Step 203a: Determine the first deviation angle between the camera and the display lens, and determine the second deviation angle between the optical waveguide structure and the display lens.

[0047] Wherein, the first deviation angle is the angle between a straight line perpendicular to the plane where the display lens is located and the axis of the center point of the camera;

[0048] The second deviation angle is the angle between a straight line perpendicular to the plane of the display lens and the axis perpendicular to the center point of the optical waveguide structure.

[0049] Step 203b: Calculate the compensation angle based on the first deviation angle and the second deviation angle.

[0050] In this embodiment of the disclosure, the compensation angle can be used to compensate for the visual deviation caused by the first deviation angle and the second deviation angle, thereby reducing the user's viewing discomfort caused by the first deviation angle and the second deviation angle during subsequent projection, improving the realism of the feedback scene image, and enhancing the user experience.

[0051] Step 203c: Adjust the projection angle of the optical waveguide structure based on the compensation angle to determine the optimal projection angle.

[0052] Step 204: Reproject and stabilize the three-dimensional image using the optical waveguide structure based on the optimal projection angle.

[0053] In summary, the augmented reality (AR) glasses and display method proposed in this application first utilize a camera to capture images of a real-world scene and send these images to an image processing unit. Then, the image processing unit performs 3D reconstruction on the real-world scene images to obtain a 3D image, which is then sent to an optical waveguide structure. Finally, the optical waveguide structure is used to reproject and stabilize the 3D image, displaying it on the display lenses of the AR glasses. By reconstructing the real-world scene images to obtain a 3D image and then reprojecting and stabilizing the reconstructed 3D image onto the lenses, the user's viewing discomfort caused by the angle discrepancy between the camera and the AR glasses lenses can be compensated for, improving the realism of the displayed scene and enhancing the user experience, thus demonstrating significant application value.

[0054] Example 3:

[0055] Figure 3 A structural diagram of an AR glasses display device 300 provided in an embodiment of this application is shown below. Figure 3 As shown, the device may include:

[0056] Camera 301 is used to capture images of real-world scenes and send the captured images to the image processing unit.

[0057] Image processing unit 302 is used to perform three-dimensional reconstruction on the real scene image to obtain a three-dimensional image, and send the three-dimensional image to the optical waveguide structure;

[0058] The optical waveguide structure 303 is used to reproject and stabilize the three-dimensional image to display the three-dimensional image on the display lens of the AR glasses.

[0059] In an embodiment of this application, the image processing unit 302 is further configured to calculate the optimal projection angle and send the optimal projection angle to the optical waveguide structure;

[0060] The optical waveguide structure 303 is also used to reproject and stabilize the three-dimensional image based on the optimal projection angle.

[0061] Furthermore, the image processing unit 302 is also used for:

[0062] Determine a first deviation angle between the camera and the display lens, and determine a second deviation angle between the optical waveguide structure and the display lens;

[0063] A compensation angle is calculated based on the first deviation angle and the second deviation angle, and the compensation angle is used to compensate for the visual deviation caused by the first deviation angle and the second deviation angle.

[0064] The optimal projection angle is determined by adjusting the projection angle of the optical waveguide structure based on the compensation angle.

[0065] In embodiments of this application, the image processing unit 302 is further configured to:

[0066] The image information of the real scene image is input into the 3D reconstruction model to obtain the corresponding 3D mesh data, wherein the 3D reconstruction model is pre-trained based on a neural network algorithm;

[0067] A three-dimensional image is obtained by performing three-dimensional reconstruction using the three-dimensional mesh data.

[0068] In the embodiments of this application, both the camera 301 and the optical waveguide structure 303 are disposed on the side of the display lens;

[0069] The camera 301 and the optical waveguide structure 303 are disposed on the same side of the display lens; or the camera and the optical waveguide structure are disposed on opposite sides of the display lens.

[0070] In summary, the augmented reality (AR) glasses proposed in this application first use a camera to capture images of the real-world scene and send these images to an image processing unit. Then, the image processing unit performs 3D reconstruction on the real-world scene images to obtain a 3D image, which is then sent to an optical waveguide structure. Finally, the optical waveguide structure is used to reproject and stabilize the 3D image, displaying it on the display lenses of the AR glasses. By reconstructing the real-world scene images to obtain a 3D image and using the optical waveguide structure to reproject and stabilize the 3D image, the realism of the displayed scene is improved, and the user experience is enhanced, demonstrating significant application value.

[0071] To implement the above embodiments, this disclosure also proposes an AR glasses.

[0072] The AR glasses provided in this disclosure include a camera, an image processing device, and an optical waveguide structure. After the AR glasses are configured to run computer instructions stored in a computer-readable storage medium, they can implement the method described in Embodiment 1.

[0073] To implement the above embodiments, this disclosure also proposes a computer-readable storage medium.

[0074] The computer-storable medium provided in this embodiment stores computer instructions, which, when executed by a processor, can implement the method described in Embodiment 1.

[0075] The following examples illustrate the specific methods of this application, based on the above content:

[0076] When a user needs to observe an image while using AR glasses, the AR glasses first use a camera to capture an image of the real scene outside and send the captured image to an image processing unit. Then, the image processing unit uses the image processing unit to perform three-dimensional reconstruction on the real scene image to obtain a three-dimensional image, and sends the three-dimensional image to an optical waveguide structure. Finally, the optical waveguide structure uses the three-dimensional image to perform reprojection and image stabilization display, so as to display the three-dimensional image on the display lens of the AR glasses.

[0077] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0078] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0079] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A method for displaying augmented reality (AR) glasses, characterized in that, The AR glasses include a camera, an image processing device, and an optical waveguide structure; the method includes: The system uses a camera to capture images of real-world scenes and then sends these images to an image processing unit. The image processing unit performs three-dimensional reconstruction on the real scene image to obtain a three-dimensional image, and then sends the three-dimensional image to the optical waveguide structure; The image processing unit calculates the optimal projection angle and sends the optimal projection angle to the optical waveguide structure. The three-dimensional image is reprojected and stabilized based on the optimal projection angle using the optical waveguide structure, so as to display the three-dimensional image on the display lens of the AR glasses; The calculation of the optimal projection angle includes: Determine a first deviation angle between the camera and the display lens, and determine a second deviation angle between the optical waveguide structure and the display lens; A compensation angle is calculated based on the first deviation angle and the second deviation angle, and the compensation angle is used to compensate for the visual deviation caused by the first deviation angle and the second deviation angle. The optimal projection angle is determined by adjusting the projection angle of the optical waveguide structure based on the compensation angle.

2. The method as described in claim 1, characterized in that, The step of using the image processing unit to perform three-dimensional reconstruction of the real scene image to obtain a three-dimensional image includes: The image information of the real scene image is input into the three-dimensional reconstruction model to obtain the corresponding three-dimensional mesh data, wherein the three-dimensional reconstruction model is pre-trained based on the neural network method; A three-dimensional image is obtained by performing three-dimensional reconstruction using the three-dimensional mesh data.

3. The method as described in claim 1, characterized in that, Both the camera and the optical waveguide structure are disposed on the side of the display lens; The camera and the optical waveguide structure are disposed on the same side of the display lens; or the camera and the optical waveguide structure are disposed on opposite sides of the display lens.

4. An AR glasses display device, characterized in that, The AR glasses include a camera, an image processing device, and an optical waveguide structure. The device includes: The camera is used to capture images of real-world scenes and send them to the image processing unit. The image processing unit is used to perform three-dimensional reconstruction on the real scene image to obtain a three-dimensional image, and send the three-dimensional image to the optical waveguide structure; calculate the optimal projection angle, and send the optimal projection angle to the optical waveguide structure. An optical waveguide structure is used to reproject and stabilize the three-dimensional image based on the optimal projection angle, so as to display the three-dimensional image on the display lens of the AR glasses; The image processing unit is further configured to: Determine a first deviation angle between the camera and the display lens, and determine a second deviation angle between the optical waveguide structure and the display lens; A compensation angle is calculated based on the first deviation angle and the second deviation angle, and the compensation angle is used to compensate for the visual deviation caused by the first deviation angle and the second deviation angle. The optimal projection angle is determined by adjusting the projection angle of the optical waveguide structure based on the compensation angle.

5. The apparatus as described in claim 4, characterized in that, The image processing unit is further configured to: The image information of the real scene image is input into the 3D reconstruction model to obtain the corresponding 3D mesh data, wherein the 3D reconstruction model is pre-trained based on a neural network algorithm; A three-dimensional image is obtained by performing three-dimensional reconstruction using the three-dimensional mesh data.

6. The apparatus as claimed in claim 4, characterized in that, Both the camera and the optical waveguide structure are disposed on the side of the display lens; The camera and the optical waveguide structure are disposed on the same side of the display lens; or the camera and the optical waveguide structure are disposed on opposite sides of the display lens.

7. An AR glasses, characterized in that, The AR glasses include a camera, an image processing device, and an optical waveguide structure, and the AR glasses are configured to run computer instructions stored in a computer-readable storage medium to perform the method described in any one of claims 1-3.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 3.

Images