Augmented reality (AR) projection method and apparatus, and ar glasses and storage medium

Abstract

An augmented reality (AR) projection method and apparatus, and AR glasses and a storage medium. The augmented reality projection method is applied to AR glasses, which comprise a projection module and a camera module, wherein extrinsic parameters between the projection module and the camera module are calibrated in real time, the projection module is configured to project a virtual picture into a three-dimensional scene, and the camera module is configured to photograph the three-dimensional scene. The method comprises: acquiring, at intervals of a pre-determined duration, a first image which is captured by means of a camera module, wherein the first image includes a virtual picture; determining actual projection coordinates of the virtual picture in the first image; and if the actual projection coordinates are inconsistent with target projection coordinates of the virtual picture in the first image, on the basis of extrinsic parameters between a projection module and the camera module, adjusting the projection module, such that the projection module projects the virtual picture to a target projection region corresponding to the target projection coordinates in a three-dimensional scene. By means of the method, the present application can realize AR without relying on a projection board, and can improve the user experience.

Description

Augmented reality projection method, device, AR glasses and storage medium

[0001] This application claims priority to a Chinese patent application filed with the Patent Office of China on March 22, 2024, with application number 202410336729X, entitled “Augmented reality projection method, device, AR glasses and storage medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The embodiments of the present application relate to the field of augmented reality technology, and specifically to an augmented reality projection method, device, AR glasses, and storage medium. Background Art

[0003] Augmented Reality (AR) technology is a technology that integrates virtual images such as text, images, three-dimensional models, music, and video into the real world, allowing virtual images and real-world information to complement each other, thereby achieving "enhancement" of the real world.

[0004] Currently, users can see virtual images by wearing AR glasses without affecting their observation of the real world. In traditional solutions, when using AR glasses to achieve augmented reality, the virtual image is usually sent to the display built into the AR glasses for display, or the virtual image is projected onto the projection board built into the AR glasses, and then projected onto the user's retina through the projection board, allowing the user to see a scene where the virtual image on the display or projection board is integrated with the real world. However, this solution is highly dependent on the display or projection board, and the display or projection board is usually located in front of the glasses lenses, which will affect the user's visual effect. Summary of the Invention

[0005] In view of the above problems, the embodiments of the present application provide an augmented reality projection method, device, AR glasses and storage medium, which can enable AR glasses to project and display virtual images on the surface of real objects in a three-dimensional scene with appropriate light intensity and a suitable projection surface. AR glasses do not need to be configured with a projection board, so as to solve the problems of high dependence on projection boards and poor user experience in the existing technology.

[0006] According to one aspect of an embodiment of the present application, an augmented reality projection method is provided, which is applied to AR glasses including a projection module and a shooting module, wherein the external parameters between the projection module and the shooting module are calibrated in real time, the projection module is used to project a virtual picture into a three-dimensional scene, and the shooting module is used to shoot the three-dimensional scene. The method includes: obtaining a first image shot by the shooting module at predetermined time intervals, wherein the first image contains the virtual picture; determining the actual projection coordinates of the virtual picture in the first image; judging whether the actual projection coordinates are consistent with the target projection coordinates of the virtual picture in the first image; if the actual projection coordinates are inconsistent with the target projection coordinates, adjusting the projection module based on the external parameters between the projection module and the shooting module so that it projects the virtual picture to a target projection area corresponding to the target projection coordinates in the first image in the three-dimensional scene.

[0007] In an optional embodiment, the method further includes: if the actual projection coordinates are consistent with the target projection coordinates, performing feature extraction on the virtual image in the first image; judging whether the virtual image projected into the three-dimensional scene is distorted based on the extracted features; if the virtual image projected into the three-dimensional scene is distorted, adjusting the projection module so that it projects the corrected virtual image into the target projection area.

[0008] In an optional manner, before obtaining the first image captured by the shooting module at predetermined time intervals, the method further includes: obtaining a second image captured by the shooting module, wherein the second image contains an interactive gesture and its corresponding target projection background; extracting the first coordinates of the target projection background corresponding to the interactive gesture in the second image in the second image; and controlling the projection module to project the virtual picture to the target projection area corresponding to the first coordinates in the three-dimensional scene based on the external parameters between the projection module and the shooting module.

[0009] In an optional manner, the feature extraction of the virtual picture in the first image includes: extracting edge information of the virtual picture in the first image; and judging whether the virtual picture projected into the three-dimensional scene is distorted based on the extracted features includes: judging whether the virtual picture projected into the three-dimensional scene is distorted based on whether the extracted edge information is consistent with preset edge information.

[0010] In an optional manner, between two adjacent acquisitions of the first image captured by the shooting module, the method further includes: inputting the actual projection coordinates of the virtual picture in the historical first image into the prediction model; predicting the coordinate changes of the virtual picture in the first image through the prediction model; based on the external parameters between the projection module and the shooting module, adjusting the projection module according to the predicted coordinate changes so that it projects the virtual picture to the predicted target projection area.

[0011] In an optional manner, between two adjacent calibrations of the external parameters between the projection module and the shooting module, the method further includes: inputting the external parameters and corresponding moments between the projection module and the shooting module calibrated each time into a preset motion model; predicting the subsequent distortion of the virtual image through the motion model; and adjusting the projection module through preset motion compensation parameters based on the predicted distortion so that it projects the corrected virtual image to the target projection area.

[0012] In an optional manner, the target projection area is an area on the target real object in the three-dimensional scene, and the first image also includes the target real object; after executing the step of acquiring the first image captured by the shooting module every predetermined time for the nth time, the method further includes: extracting the second coordinates of the target real object in the first image acquired for the n-1th time in the first image, where n is a positive integer and n>1; extracting the third coordinates of the target real object in the first image acquired for the nth time in the first image; if the difference between the third coordinate and the second coordinate is greater than a preset difference, updating the target projection coordinates to the third coordinates.

[0013] According to another aspect of an embodiment of the present application, an augmented reality projection device is provided, which is applied to AR glasses including a projection module and a shooting module, wherein the external parameters between the projection module and the shooting module are calibrated in real time, the projection module is used to project a virtual picture into a three-dimensional scene, and the shooting module is used to shoot the three-dimensional scene. The device includes: an acquisition module, used to acquire a first image shot by the shooting module at predetermined intervals, wherein the first image contains the virtual picture; a determination module, used to determine the actual projection coordinates of the virtual picture in the first image; a judgment module, used to judge whether the actual projection coordinates are consistent with the target projection coordinates of the virtual picture in the first image; and an adjustment module, used to adjust the projection module based on the external parameters between the projection module and the shooting module so that it projects the virtual picture into a target projection area corresponding to the target projection coordinates in the first image in the three-dimensional scene if the actual projection coordinates are inconsistent with the target projection coordinates.

[0014] According to another aspect of an embodiment of the present application, there is provided an AR glasses, comprising: a projection module, a shooting module, a processor and a memory; real-time calibration of external parameters between the projection module and the shooting module, the projection module being used to project a virtual image into a three-dimensional scene, and the shooting module being used to shoot the three-dimensional scene; the memory being used to store executable instructions, the executable instructions enabling the processor to perform the operations of the augmented reality projection method as described above.

[0015] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which executable instructions are stored. The executable instructions, when run, execute the operations of the augmented reality projection method as described above.

[0016] The above description is only an overview of the technical solutions of the embodiments of the present application. In order to more clearly understand the technical means of the embodiments of the present application, they can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present application. In addition, the same reference symbols are used to represent the same components throughout the drawings. In the drawings:

[0018] FIG1 is a schematic diagram showing a flow chart of an augmented reality projection method provided in an embodiment of the present application;

[0019] FIG2 is a schematic diagram showing a flow chart of an augmented reality projection method provided by another embodiment of the present application;

[0020] FIG3 shows a schematic structural diagram of an augmented reality projection device provided in an embodiment of the present application;

[0021] FIG4 shows a schematic structural diagram of the AR glasses provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0023] The inventors of this application discovered that in order to enable users to experience augmented reality through AR glasses, they can generally be equipped with a projection module and a projection board, where the projection board is located in front of the glasses' lenses. The principle of this implementation is that the projection module projects a virtual image onto the projection board, which then reflects it to the human eye. Simultaneously, light reflected from the surfaces of real objects in the real world passes through the projection board and the lenses of the AR glasses into the human eye, allowing the user to see a combined virtual and real image.

[0024] However, the aforementioned augmented reality method requires the use of a projection board. Furthermore, since the projection board is located in front of the eyeglass lenses, when the user observes the real world through the eyeglass lenses, the projection board will have a certain impact on the visual effect, thereby reducing the user experience.

[0025] The inventors of this application have discovered that projecting a virtual image directly into a target projection area within a three-dimensional scene can achieve augmented reality without requiring a projection board. However, when a user wears AR glasses, their body inevitably moves, which in turn causes the position of the projection module in the AR glasses to change accordingly, causing the virtual image projected by the projection module to deviate from the target projection area.

[0026] Therefore, based on the above considerations, in order to reduce dependence on the display, improve user experience, and ensure that the virtual picture projected by the projection module is within the target projection area when realizing augmented reality, the present application proposes an augmented reality projection method. Among them, the augmented reality projection method is applied to AR glasses including a projection module and a shooting module, the projection module is used to project the virtual picture into the three-dimensional scene, and the shooting module is used to shoot the three-dimensional scene. The augmented reality projection method realizes augmented reality by projecting the virtual picture into the three-dimensional scene by using the projection module, so that the virtual picture is displayed on the surface of the real object in the three-dimensional scene. In addition, the method also determines whether the virtual picture projected by the projection module is within the target projection area based on the coordinates of the virtual picture in the image captured by the shooting module. If it is not within the target projection area, the projection module is adjusted so that the virtual picture projected by the projection module is within the target projection area.

[0027] It should be noted that AR glasses using the augmented reality projection method provided in the embodiments of the present application are suitable for use in environments where the lighting conditions meet the projection requirements and there is a suitable surface of a real object for projection in the three-dimensional scene, so that the AR glasses can project the virtual image onto the surface of the real object in the three-dimensional scene. For example, indoors or outdoors where the lighting conditions meet the projection requirements, that is, in an environment with relatively low light intensity, the user can cause the AR glasses to project the virtual image onto a white wall, table, floor, square object, etc. that is suitable for projection, or onto other projectable parts of the user, such as the palm, back of the hand, or leg, thereby achieving the fusion of the virtual image and the real object.

[0028] FIG1 shows a flow chart of an augmented reality projection method provided in an embodiment of the present application, which is applied to AR glasses comprising a projection module and a shooting module. The external parameters between the projection module and the shooting module are calibrated in real time, the projection module is used to project a virtual image into a three-dimensional scene, and the shooting module is used to shoot the three-dimensional scene. The AR glasses may be AR glasses comprising one or more processors, which may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement an embodiment of the present application, which is not limited here. The one or more processors included in the AR glasses may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs, which are not limited here. As shown in FIG1 , the method comprises the following steps:

[0029] Step 110: Obtain a first image captured by the shooting module at predetermined time intervals, wherein the first image includes a virtual screen.

[0030] The predetermined duration can be set as needed, for example, to 0.1s, 0.3s, or 0.5s. As previously mentioned, if the user moves, causing the position of the projection module to change, the virtual image projected by the projection module may deviate from the target projection area. Therefore, in this step, the first image is acquired to further determine whether the virtual image projected by the projection module is within the target projection area.

[0031] If the preset time is too long, the user may need to wait for a while after finishing moving within a shorter period of time. Only after the preset time is reached can the AR glasses obtain the first image and adjust the projection module. In this case, during the user's movement and waiting period, the virtual image projected into the three-dimensional scene will deviate from the target projection area.

[0032] Therefore, in order to avoid the above situation, when setting the predetermined time, it is preferably set to a shorter time, so that when the user moves, the first image can be obtained in time and the projection module can be adjusted, so that the virtual picture projected by the projection module into the three-dimensional scene is always located in the target projection area as much as possible, thereby improving the user experience.

[0033] In some embodiments, to avoid wasting resources, that is, when the camera module captures multiple images and only processes one of them, the interval between the camera module capturing the 3D scene can be set to the predetermined duration. It should be noted that the first image captured at each predetermined duration is the most recent image captured by the camera module.

[0034] After the projection module projects the virtual image onto a real object in the three-dimensional scene, the shooting module is used to shoot the three-dimensional scene. If the real object is within the shooting range, the virtual image is presented in the first image taken by the shooting module in the form of two-dimensional pixel points, that is, the first image contains the virtual image.

[0035] Step 120: Determine the actual projection coordinates of the virtual frame in the first image.

[0036] The actual projection coordinates are the pixel coordinates of the virtual image in the first image. After determining the actual projection coordinates, the actual projection coordinates can be further used to determine whether the position of the virtual image projected by the projection module into the three-dimensional scene is correct.

[0037] The virtual image in the first image corresponds to multiple pixels in the first image. To improve the accuracy of the judgment, the actual projection coordinates of all pixels corresponding to the virtual image in the first image can be obtained. To improve the efficiency of the judgment, the actual projection coordinates of some pixels corresponding to the virtual image in the first image can be obtained, or after setting a reference point for the virtual image, the actual projection coordinates of the pixels corresponding to the reference point in the first image can be obtained.

[0038] Step 130: Determine whether the actual projection coordinates are consistent with the target projection coordinates of the virtual screen in the first image. If not, go to step 140; if yes, end this process.

[0039] Among them, it should be noted that the world coordinates of the area to which the virtual image is desired to be projected in the real three-dimensional scene, and the corresponding pixel coordinates in the first image are the target projection coordinates. The target projection coordinates in the first image can be pre-set or obtained through analysis and processing, and are not limited here. For an image captured of a three-dimensional scene, different positions in the three-dimensional scene correspond to different pixel points in the image, and the coordinates of different pixel points in the image are different. Therefore, by judging whether the actual projection coordinates are consistent with the target projection coordinates, it can be judged whether the actual projection area where the projection module projects the virtual image into the three-dimensional scene is the target projection area. The target projection area is the area corresponding to the target projection coordinates in the three-dimensional scene.

[0040] Step 140: Based on the external parameters between the projection module and the camera module, adjust the projection module to project the virtual image to a target projection area corresponding to the target projection coordinates in the first image in the three-dimensional scene.

[0041] The extrinsic parameters between the projection module and the camera module refer to the parameters used to represent the spatial relationship between the projection module coordinate system and the camera module coordinate system. The projection module coordinate system is a coordinate system constructed with a point in the projection module as the coordinate origin. The camera module coordinate system is a coordinate system constructed with a point in the camera module as the coordinate origin. Using the extrinsic parameters between the projection module and the camera module, the projection module coordinate system and the camera module coordinate system can be converted.

[0042] Real-time calibration of the extrinsic parameters between the projection module and the camera module involves timely calibration and updating of the extrinsic parameters between the projection module and the camera module whenever the spatial relationship between them changes, such as when the projection module's position changes. This step and all subsequent steps that adjust the projection module and the camera module based on the extrinsic parameters between them are based on the most recently updated extrinsic parameters between the projection module and the camera module.

[0043] Given the actual projection coordinates of the virtual image in the first image and the external parameters between the projection module and the shooting module, if one wants to project the virtual image to the area corresponding to the target projection coordinates in the first image in the three-dimensional scene, it is necessary to determine how the process of changing from the actual projection coordinates to the target projection coordinates in the first image corresponds to the movement of the projection module. Since the external parameters between the projection module and the shooting module are known, it is only necessary to calculate the change from the actual projection coordinates to the target projection coordinates in the first image based on the external parameters between the projection module and the shooting module, and then convert the change into the change in the projection module coordinate system. The projection module can be adjusted with the obtained change in the projection module coordinate system to enable the projection module to project the virtual image to the target projection area in the three-dimensional scene.

[0044] In the embodiments of the present application, the projection module in the AR glasses is used to project virtual images onto real objects in a three-dimensional scene, thereby integrating the real objects with the virtual images, thereby achieving augmented reality without the need for a display or projection board. Furthermore, since the AR glasses in the embodiments of the present application do not have a display or projection board, when the user uses the AR glasses to observe the real world, there will be no situation where the display or projection board blocks or affects the user's observation field of view, thereby improving the user experience.

[0045] It is understandable that the target projection area is the area on the target real object in the three-dimensional scene. When realizing augmented reality, it is necessary to merge the virtual picture with the target real object to have practical significance, rather than merging the virtual picture with any real object. Therefore, in the embodiment of the present application, by judging whether the actual projection coordinates are consistent with the target projection coordinates, and adjusting the posture of the projection module in time according to the judgment result, so that the picture projected by the projection module is always within the target projection area as much as possible, there will basically be no situation where the projection module posture changes due to the movement of the user's head when wearing AR glasses, causing the virtual picture projected by the projection module to deviate from the target projection area.

[0046] In some embodiments, if the coordinates of the target projection area in the three-dimensional scene are Pt, the external parameters between the projection module and the shooting module are Tproj_cam, that is, the parameters for converting the shooting module coordinate system to the projection module coordinate system are Tproj_cam, and the actual coordinates of the virtual image projected by the projection module to the three-dimensional scene are Pp, then the actual coordinates Pp can be calculated using the following formula (1): Pp = Tproj_cam*Pt (1).

[0047] In some embodiments, to achieve real-time calibration of external parameters between the projection module and the camera module, calibration frames can be added in real time, allowing real-time detection of the calibrated 3D geometric information of the projection module. The acquisition frequency can be set to 60 Hz. Since the human eye only pays attention to 25 Hz video, a frame for real-time projection module and camera module plane detection can be superimposed on a higher frequency frame without affecting the user experience.

[0048] In order to improve the display effect of the virtual image projected onto the target projection area, FIG2 shows a flow chart of an augmented reality projection method provided by another embodiment of the present application, based on the embodiment provided in FIG1 . As shown in FIG2 , the method includes the following steps:

[0049] Step 110: Obtain a first image captured by the shooting module at predetermined time intervals, wherein the first image includes a virtual screen.

[0050] Step 120: Determine the actual projection coordinates of the virtual frame in the first image.

[0051] Step 130: Determine whether the actual projection coordinates are consistent with the target projection coordinates of the virtual screen in the first image. If not, go to step 140; if so, go to step 150.

[0052] Step 140: Based on the external parameters between the projection module and the camera module, adjust the projection module to project the virtual image to a target projection area corresponding to the target projection coordinates in the first image in the three-dimensional scene.

[0053] The principles and specific implementations of steps 110 to 140 may refer to the embodiment provided in FIG1 , and will not be described in detail here.

[0054] Step 150: Extract features of the virtual image in the first image.

[0055] Particularly, certain characters in the virtual picture in the first image may be extracted.

[0056] Step 160: Determine whether the virtual image projected into the three-dimensional scene is distorted based on the extracted features. If so, go to step 170; if not, end this process.

[0057] If the virtual image projected into the 3D scene is distorted, the characters in the virtual image will also be distorted accordingly. Therefore, by determining whether the extracted characters are distorted, it is possible to determine whether the virtual image projected into the 3D scene is distorted.

[0058] Step 170: Adjust the projection module to project the corrected virtual image onto the target projection area.

[0059] If the virtual image is distorted, the actual coordinates of the virtual image projected into the 3D scene will not match the target coordinates. Therefore, by adjusting the position of the projection module so that it projects the corrected virtual image to the target coordinates, the virtual image projected into the 3D scene can be undistorted.

[0060] In order to ensure that the projection module can project the corrected virtual image to the target projection area, a method for determining the coordinates of the corrected virtual image is given here. Distortion includes radial distortion and tangential distortion. For radial distortion, the horizontal coordinate of the corrected virtual image can be calculated using the following formula (2), and the vertical coordinate of the corrected virtual image can be calculated using the following formula (3): x'=x(1+k1*r 2 +k2*r 4 +k3*r 6) (2); y'=y(1+k1*r 2 +k2*r 4 +k3*r 6 ) (3);

[0061] Among them, x' and y' are the horizontal and vertical coordinates of the corrected virtual image, respectively, x and y are the actual horizontal and vertical coordinates of the observed virtual image, respectively, r 2 =x 2 +y 2 , k1, k2 and k3 are radial distortion parameters and can be set as needed.

[0062] For tangential distortion, the horizontal coordinate of the corrected virtual image can be calculated using the following formula (4), and the vertical coordinate of the corrected virtual image can be calculated using the following formula (5): x" = x + (2p1*x*y + p2(r 2 +2x 2 )) (4); y"=y+(p1(r 2 +2y 2 )+2p2*x*y) (5);

[0063] Wherein, x' and y' are the horizontal and vertical coordinates of the corrected virtual image, respectively; x' and y' are the actual horizontal and vertical coordinates of the observed virtual image, respectively; p1 and p2 are tangential distortion parameters, which can be set as needed.

[0064] If the virtual image projected into the three-dimensional scene is distorted, the user will not be able to effectively access the virtual image content, reducing the effectiveness of augmented reality. Therefore, in the embodiments of the present application, by determining whether the virtual image projected into the three-dimensional scene is distorted and adjusting the projection module in a timely manner based on the determination result, the projection module projects the corrected virtual image into the three-dimensional scene, thereby improving the effectiveness of augmented reality.

[0065] In some embodiments, the virtual image can be projected to the actual coordinates Pp in the three-dimensional scene based on the coordinates Pt of the target projection area in the three-dimensional scene and the projection module to calculate the distortion of the virtual image, for example, changes in the shape and position of the virtual image.

[0066] In some embodiments, if the coordinates of the corrected virtual frame are Pp_corrected, Pp_corrected can be calculated using the following formula (6): Pp_corrected=Tproj_cam*Pt (6).

[0067] In order to improve the user experience, the virtual image is projected to the target projection area specified by the user. Based on the embodiment provided in FIG. 1 , in the embodiment of the present application, before step 110, the method further includes the following steps:

[0068] Step a1: Acquire a second image captured by the shooting module, wherein the second image includes an interactive gesture and its corresponding target projection background.

[0069] Among them, in an embodiment of the present application, in order to determine the target projection area and improve the interaction effect between the AR glasses and the user, the area in the three-dimensional scene pointed to by the user through the interactive gesture is determined as the target projection area. After the user points to the target projection area through the interactive gesture, the second image obtained by capturing the user's interactive gesture and the target area through the shooting module, the interactive gesture and the target projection area are presented in the second image in the form of two-dimensional pixels. That is, the second image contains the interactive gesture and its corresponding target projection background, which refers to the pixel points corresponding to the interactive gesture and the target projection area in the second image.

[0070] Step a2: extracting the first coordinate of the target projection background corresponding to the interactive gesture in the second image in the second image.

[0071] Among them, this step is similar to step 120. Therefore, the specific implementation method of this step can refer to step 120 and will not be repeated here.

[0072] Step a3: Based on the external parameters between the projection module and the shooting module, control the projection module to project the virtual image to a target projection area corresponding to the first coordinate in the three-dimensional scene.

[0073] Among them, this step is similar to step 140. Therefore, the specific implementation method of this step can refer to step 140 and will not be repeated here.

[0074] In the embodiments of the present application, by determining the area corresponding to the interactive gesture as the target projection area, the virtual image can be projected to the area specified by the user according to the user's needs, thereby meeting the user's personalized needs. Furthermore, the user can specify the target projection area by interacting with the AR glasses, which is simple to operate. Compared with the method of requiring the user to input the target projection area in the AR glasses, the operation time can be reduced, thereby improving the efficiency of determining the target projection area.

[0075] In order to improve the efficiency of determining whether the projected virtual picture is distorted, based on the embodiment provided in FIG. 2 , step 150 includes: extracting edge information of the virtual picture in the first image.

[0076] The edge information of the virtual image in the first image refers to the outline of the virtual image in the first image. It is understood that when a projection module is used to capture a three-dimensional scene, the range of the capture is wider than the range observed by the AR glasses. Therefore, the first image does not only contain the virtual image. Based on this, the outline of the virtual image in the first image can be extracted from the first image.

[0077] Step 160 includes determining whether the virtual image projected into the three-dimensional scene is distorted based on whether the extracted edge information is consistent with the preset edge information.

[0078] When a virtual image projected into a three-dimensional scene is distorted, its outline typically changes. For example, if the outline of the undistorted virtual image is rectangular, after distortion, its outline may become trapezoidal. The preset edge information can be set as needed. For example, if the outline of the undistorted virtual image is rectangular, the preset edge information can be set to a rectangular shape. Therefore, if the extracted edge information is inconsistent with the preset edge information, it can be determined that the virtual image projected into the three-dimensional scene is distorted.

[0079] In an embodiment of the present application, the method of determining whether the virtual image is distorted is performed by comparing the edge information of the virtual image projected in the three-dimensional scene with the preset edge information. The determination method is simple and time-saving, thereby improving the efficiency of determining whether the virtual image is distorted.

[0080] To further improve user experience, based on the embodiment provided in FIG. 1 , in an embodiment of the present application, between two adjacent acquisitions of the first image captured by the camera module, the method further includes the following steps:

[0081] Step b1: input the actual projection coordinates of the virtual screen in the first historical image into the prediction model.

[0082] Among them, the actual projection coordinates of the virtual screen in each of the multiple first images acquired in the recent period can be input into the prediction model, and then the prediction model can be used to predict the change in the relative position between the target real object and the AR glasses at the next moment.

[0083] Step b2: Predicting the coordinate changes of the virtual picture in the first image using a prediction model.

[0084] Among them, the prediction model can predict the relative position change between the target real object and the AR glasses at the next moment based on the actual projection coordinates of the virtual picture in the first image obtained during the input historical period, that is, predict the coordinate change of the virtual picture in the first image.

[0085] For example, if the target real object is stationary and the user continues to move to the right, causing the relative position between the target real object and the AR glasses to change, the prediction model can predict that the user will move to the right at the next moment, that is, the user's position at the next moment, based on the coordinates of the virtual picture in the first image obtained during the historical period in the first image, and thus predict the coordinate changes of the virtual picture in the first image at the next moment.

[0086] Step b3: Based on the external parameters between the projection module and the shooting module, and according to the predicted coordinate changes, adjust the projection module so that it projects the virtual image to the predicted target projection area.

[0087] The predicted target projection area, ie, the target projection area at the next moment, can be determined by the coordinate change determined in step b2.

[0088] If the relative position between the AR glasses and the target real object changes, the coordinates of the virtual image in the first image will also change. If the position of the projection module is not adjusted in time, the virtual image projected by the projection module will deviate from the target real object.

[0089] Therefore, in the embodiment of the present application, when the relative position between the target real object and the AR glasses changes, the change in the relative position between the target real object and the AR glasses at the next moment is predicted, and the position of the projection module is adjusted in a timely manner so that the virtual image projected by the projection module is always on the target real object. By tracking the position changes between the target real object and the AR glasses in real time and dynamically adjusting the target projection area, it is possible to adapt to the movement of at least one of the target real object and the AR glasses, resulting in changes in the relative position between the two, or changes in the three-dimensional scene.

[0090] Furthermore, by adjusting the projection module's position in advance, the virtual image the user sees through the AR glasses will already be projected onto the target real object when the relative position between the target real object and the AR glasses changes. This reduces user waiting time compared to waiting for the relative position between the target real object and the AR glasses to change, and then taking some time to adjust the projection module's position accordingly.

[0091] In some embodiments, an adaptive target projection area adjustment algorithm can be used to adjust the target projection area according to factors such as the size, position, and movement speed of the target real object to ensure that the virtual image projected by the projection module is always located on the target real object.

[0092] In order to improve the display effect of the virtual screen, based on the embodiment provided in FIG. 1 , in the embodiment of the present application, between two adjacent calibrations of the external parameters between the projection module and the camera module, the method further includes the following steps:

[0093] Step c1: Input the external parameters and corresponding moments between the projection module and the shooting module calibrated each time into a preset motion model.

[0094] Step c2: Predict the distortion of the virtual image using a motion model.

[0095] Among them, since the virtual image projected into the three-dimensional scene is distorted, the position of the projection module is adjusted so that it can project the corrected virtual image into the three-dimensional scene. When the position of the projection module changes, the external parameters between the projection module and the camera module are calibrated in a timely manner. Therefore, in steps c1 and c2, by inputting the external parameters and the corresponding time of each calibration into the preset motion model, the preset motion model uses the distortion of the virtual image in the historical period to predict the distortion of the virtual image at the next moment, so that the correction process can be performed in advance.

[0096] Step c3: According to the predicted distortion, the projection module is adjusted using preset motion compensation parameters so that the projected virtual image is projected onto the target projection area.

[0097] If the relative position between the AR glasses and the target real object changes, the virtual image projected into the 3D scene may be distorted. Therefore, in the embodiment of the present application, the motion model is used to estimate the distortion of the virtual image at the next moment based on the distortion of the virtual image at the previous moment, and then compensate for it during the correction process, thus performing the correction process in advance.

[0098] In some embodiments, assuming that at time t, the external parameter between the shooting module and the projection module is Tproj_cam(t), the coordinates of the target real object in the camera coordinate system are Pt(t), and the coordinates of the virtual image are Pp(t), the motion model processing process can be expressed as: Tproj_cam(t)=f(Tproj_cam(t-1), Δt);

[0099] Where f is the motion model function and Δt is the time interval.

[0100] During the distortion correction process, the motion compensation parameter C(t) is used to compensate for the distortion of the virtual image projected onto the 3D scene caused by the movement of the AR glasses. Assuming that the corrected virtual image coordinates are Pp_corrected(t), the distortion correction process can be expressed as: Pp_corrected(t) = Pp(t) + C(t);

[0101] Wherein, C(t) is a motion supplement parameter used to compensate for the distortion of the virtual image projected into the three-dimensional scene caused by the motion of the projection module and the camera module within the time interval Δt.

[0102] In some embodiments, if the projection module and the shooting module cause the projected virtual image to be distorted during movement, when using correction parameters to correct the distorted virtual image, the correction parameters are dynamically adjusted according to the movement state of the projection module and the shooting module, thereby ensuring that the virtual image projected into the three-dimensional scene is not distorted.

[0103] In some embodiments, by using technologies such as visual odometry, the external parameters between the projection module and the shooting module at the next moment are estimated in real time, and then the projection module is adjusted according to the estimated external parameters, so that when the three-dimensional scene changes, the changes in the three-dimensional scene can be adapted.

[0104] In some embodiments, a motion compensation algorithm is introduced to offset the distortion of the virtual image projected into the three-dimensional scene caused by the movement of the shooting module and the projection module.

[0105] In order to ensure that the virtual image is always projected on the target real object, wherein the target projection area is the area on the target real object in the three-dimensional scene. Based on the embodiment provided in FIG1 , in the embodiment of the present application, the first image also includes the target real object. After executing step 110 for the nth time, the method further includes the following steps:

[0106] Step c1: extracting the second coordinates of the target real object in the first image acquired for the n-1th time, where n is a positive integer and n>1.

[0107] Among them, in the first image captured by the shooting module of the target real object in the three-dimensional scene, the target real object is presented in the first image in the form of two-dimensional pixel points, that is, the first image contains the target real object.

[0108] Step c2: extracting the third coordinate of the target real object in the first image acquired for the nth time.

[0109] The first image acquired for the nth time refers to the most recently acquired first image. The time when the first image is acquired for the (n-1)th time is earlier than the time when the first image is acquired for the nth time.

[0110] Step c1 and step c2 are similar to step 120. Therefore, the specific implementation of step c1 and step c2 can refer to the implementation of step 120 and will not be repeated here.

[0111] Step c3: If the difference between the third coordinate and the second coordinate is greater than a preset difference, the target projection coordinate is updated to the third coordinate.

[0112] If the target real object in the three-dimensional scene has not moved, the second coordinate and the third coordinate are consistent. If the target real object has moved, the second coordinate and the third coordinate are inconsistent. The difference between the third coordinate and the second coordinate is related to the distance the target real object has moved. The greater the distance the target real object has moved, the greater the difference between the third coordinate and the second coordinate. The preset difference value can be set as needed.

[0113] As mentioned earlier, achieving augmented reality requires integrating the virtual image with the target real-world object to achieve practical results. If the target real-world object moves but the virtual image does not move accordingly, the virtual image projected by the projection module will deviate from the target real-world object, failing to achieve the desired augmented reality.

[0114] Therefore, in an embodiment of the present application, if the difference between the third coordinate and the second coordinate is greater than a preset difference, it can be determined that the target real object has moved to a certain extent, and the target projection coordinates are then updated to the third coordinates. When steps 120 to 130 are subsequently executed, it is determined whether the actual projection coordinates in the first image acquired for the nth time are consistent with the third coordinates. If they are inconsistent, the position of the projection module is adjusted so that the projection module projects the virtual image to the target projection area corresponding to the third coordinate in the three-dimensional scene. In this way, when the target real object moves, the virtual image projected into the three-dimensional scene also moves accordingly, thereby ensuring that the virtual image is always projected on the target real object.

[0115] Figure 3 shows a schematic diagram of the structure of an augmented reality projection device provided in an embodiment of the present application. This augmented reality projection device is applied to AR glasses comprising a projection module and a camera module. External parameters between the projection module and the camera module are calibrated in real time. The projection module is used to project a virtual image into a three-dimensional scene, and the camera module is used to capture the three-dimensional scene. As shown in Figure 3, the device 200 comprises an acquisition module 201, a determination module 202, a judgment module 203, and an adjustment module 204. The acquisition module 201 is used to acquire a first image captured by the camera module at predetermined intervals, wherein the first image includes a virtual image. The determination module 202 is used to determine the actual projection coordinates of the virtual image in the first image. The judgment module 203 is used to determine whether the actual projection coordinates are consistent with the target projection coordinates of the virtual image in the first image. The adjustment module 204 is used to adjust the projection module, based on the external parameters between the projection module and the camera module, to project the virtual image into a target projection area in the three-dimensional scene corresponding to the target projection coordinates in the first image.

[0116] The augmented reality projection device provided in this embodiment is used to implement the technical solution of the augmented reality projection method in the aforementioned method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0117] It is worth noting that the augmented reality projection device provided in this embodiment also includes other modules for executing the steps of the above-mentioned augmented reality projection method embodiment, which will not be described one by one here.

[0118] FIG4 shows a schematic structural diagram of the AR glasses provided in an embodiment of the present application. The specific embodiments of the present application do not limit the specific implementation of the AR glasses.

[0119] As shown in FIG. 4 , the AR glasses may include: a shooting module 302 , a projection module 304 , a processor 306 and a memory 308 .

[0120] Among them: the external parameters between the shooting module 302 and the projection module 304 are calibrated in real time, the projection module 304 is used to project the virtual image into the three-dimensional scene, and the shooting module 302 is used to shoot the three-dimensional scene.

[0121] The processor 306 is configured to execute the program 310 , and specifically may execute the relevant steps in the above-mentioned embodiment of the method for augmented reality projection.

[0122] Specifically, the program 310 may include program code including computer-executable instructions.

[0123] The processor 306 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the AR glasses may be processors of the same type, such as one or more CPUs, or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0124] The memory 308 is used to store the program 310. The memory 308 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0125] The shooting module 302 and the projection module 304 are built into the AR glasses, and the built-in projection module 302 is directly integrated into the structure of the AR glasses, so that the AR glasses are portable and convenient for users to wear and obtain an augmented reality experience. The projection module 304 does not need to project the virtual image onto the lenses in the AR glasses, but directly projects it onto the surface of objects in the three-dimensional scene, so that users can have a more extensive and immersive AR experience. In addition, the shooting module 302 adopts advanced real-time three-dimensional recognition technology, which can capture three-dimensional information of the surrounding environment and identify objects. The identified object information is superimposed and displayed with the virtual image, allowing users to obtain relevant information or interact with the identified objects in real time.

[0126] In some embodiments, the projection module 304 is configured to project a specifically encoded grating or lattice pattern onto the surface of the target real object. The camera module 302 can capture the distortion of these patterns on the surface of the target real object and infer the three-dimensional structure of the surface of the target real object based on the distortion information.

[0127] In other embodiments, AR glasses include multiple camera modules that simultaneously capture virtual images projected onto the surface of a target real object. The three-dimensional structure of the target real object's surface is inferred using parallax information between the images captured by the multiple projection modules. The virtual images projected by the projection modules can be used to improve feature point detection or matching.

[0128] In other embodiments, a sensor with depth perception capabilities can be used in conjunction with a projection module to obtain depth information about the surface of a target real object. The sensor with depth perception capabilities can be a Time-of-Flight (ToF) camera, a structured light camera, or a LiDAR. The projection module is used to project specific optical patterns for depth sensing, and the camera module captures these patterns and infers the three-dimensional shape of the target real object's surface.

[0129] An embodiment of the present application provides a computer-readable storage medium storing executable instructions. When the executable instructions are executed on AR glasses, the AR glasses execute the augmented reality projection method in any of the above method embodiments.

[0130] An embodiment of the present application provides a computer program that can be called by a processor to enable AR glasses to execute the augmented reality projection method in any of the above method embodiments.

[0131] An embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed on a computer, the computer executes the augmented reality projection method in any of the above method embodiments.

[0132] In the several embodiments provided in this application, if any function is implemented in the form of a software function module / unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the technical solution of this application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server or other electronic device) to execute all or part of the steps of the method described in each embodiment of this application. The aforementioned storage medium includes: various media that can store computer program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0133] The algorithm or demonstration provided here are not inherently relevant to any particular computer, virtual system or other equipment. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing this type of system. In addition, the present application embodiment is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the present application described here, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the present application.

[0134] It should be noted that the above embodiments illustrate rather than limit the present application, and that a person skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application may be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In claims that list several means, several units or modules of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments should not be understood as limiting the order of execution unless otherwise specified.

[0135] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. An augmented reality projection method, characterized in that: The method is applied to AR glasses including a projection module and a camera module, wherein the external parameters between the projection module and the camera module are calibrated in real time, the projection module is used to project a virtual image into a three-dimensional scene, and the camera module is used to shoot the three-dimensional scene. The method includes: acquiring a first image captured by the shooting module at predetermined intervals, wherein the first image includes the virtual screen; determining actual projection coordinates of the virtual picture in the first image; Determining whether the actual projection coordinates are consistent with the target projection coordinates of the virtual picture in the first image; If the actual projection coordinates are inconsistent with the target projection coordinates, the projection module is adjusted based on the external parameters between the projection module and the shooting module so that it projects the virtual image to the target projection area corresponding to the target projection coordinates in the first image in the three-dimensional scene.

2. The augmented reality projection method according to claim 1, characterized in that: The method further comprises: If the actual projection coordinates are consistent with the target projection coordinates, performing feature extraction on the virtual picture in the first image; Determine whether the virtual image projected into the three-dimensional scene is distorted based on the extracted features; If the virtual image projected into the three-dimensional scene is distorted, the projection module is adjusted to project the corrected virtual image into the target projection area.

3. The augmented reality projection method according to claim 1, wherein: Before acquiring the first image captured by the capture module at predetermined intervals, the method further includes: Acquire a second image captured by the capture module, wherein the second image includes an interactive gesture and a corresponding target projection background; Extracting first coordinates of a target projection background corresponding to the interactive gesture in the second image in the second image; Based on the external parameters between the projection module and the shooting module, the projection module is controlled to project the virtual image to a target projection area corresponding to the first coordinate in the three-dimensional scene.

4. The augmented reality projection method according to claim 2, wherein: The extracting features of the virtual picture in the first image includes: extracting edge information of the virtual picture in the first image; The determining, based on the extracted features, whether the virtual image projected into the three-dimensional scene is distorted includes: Whether the virtual image projected into the three-dimensional scene is distorted is determined based on whether the extracted edge information is consistent with the preset edge information.

5. The augmented reality projection method according to claim 1, wherein: Between two adjacent acquisitions of the first image captured by the shooting module, the method further includes: Inputting the actual projection coordinates of the virtual frame in the first historical image into the prediction model; Predicting the coordinate change of the virtual picture in the first image by using the prediction model; Based on the external parameters between the projection module and the shooting module, and according to the predicted coordinate changes, the projection module is adjusted to project the virtual image onto the predicted target projection area.

6. The augmented reality projection method according to claim 1, wherein: Between two adjacent calibrations of the external parameters between the projection module and the shooting module, the method further includes: Inputting the external parameters and corresponding moments between the projection module and the shooting module calibrated each time into a preset motion model; Predicting the subsequent distortion of the virtual image using the motion model; According to the predicted distortion, the projection module is adjusted by using preset motion compensation parameters so that the projection module projects the corrected virtual image onto the target projection area.

7. The method according to claim 1, characterized in that The target projection area is an area on a target real object in a three-dimensional scene, and the first image also includes the target real object; After the step of acquiring the first image captured by the capture module every predetermined time period is performed for the nth time, the method further includes: Extracting the second coordinates of the target real object in the first image acquired for the n-1th time, where n is a positive integer and n>1; Extracting the third coordinate of the target real object in the first image acquired for the nth time; If the difference between the third coordinate and the second coordinate is greater than a preset difference, the target projection coordinate is updated to the third coordinate.

8. An augmented reality projection device, characterized in that: Applicable to AR glasses including a projection module and a camera module, wherein the external parameters between the projection module and the camera module are calibrated in real time, the projection module is used to project a virtual image into a three-dimensional scene, and the camera module is used to capture the three-dimensional scene, the device comprising: an acquisition module, configured to acquire, at predetermined intervals, a first image captured by the capture module, wherein the first image includes the virtual screen; a determination module, configured to determine actual projection coordinates of the virtual picture in the first image; a judging module, configured to judge whether the actual projection coordinates are consistent with the target projection coordinates of the virtual picture in the first image; An adjustment module is used to adjust the projection module so that it projects the virtual image to the target projection area corresponding to the target projection coordinates in the first image in the three-dimensional scene based on the external parameters between the projection module and the shooting module if the actual projection coordinates are inconsistent with the target projection coordinates.

9. AR glasses, characterized in that: include: Projection module, camera module, processor and memory; Real-time calibration of external parameters between the projection module and the shooting module, the projection module is used to project a virtual image into a three-dimensional scene, and the shooting module is used to shoot the three-dimensional scene; The memory is used to store executable instructions, and the executable instructions enable the processor to execute the augmented reality projection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The storage medium stores executable instructions, and the executable instructions execute the augmented reality projection method according to any one of claims 1 to 7 when running.

Images