AR model pose display continuous correction method, device and storage device

Abstract

The application provides an AR model pose display continuous correction method, equipment and a storage device, the method comprises the following steps: connecting an AR device with a high-precision GNSS device, initializing the equipment and judging the GNSS device state, starting a camera, recognizing and extracting feature points by using an ORB algorithm and obtaining AR coordinates of the feature points, calculating an average conversion relationship matrix of a plane geographic coordinate system and an AR coordinate system, calculating the pose of an object in the AR scene when matching and displaying according to the average conversion relationship matrix, adjusting the AR model in the picture, and calculating the plane geographic coordinates of the feature points when the equipment is normal, updating the feature information and the geographic coordinates of the feature points. The equipment and the storage device are used to realize the method, and the application has the beneficial effects that the display and continuous correction of the AR model pose are realized, the accuracy of the matching between the AR scene and the reality is ensured, and the plane geographic coordinates are calculated multiple times to take the average value, so that the plane geographic coordinate accuracy of the feature points is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of AR, in particular to an AR model pose display continuous correction method, device and storage device. BACKGROUND

[0002] With the development of technology, especially the development of intelligent mobile devices, the application of augmented reality (AR) technology is more and more. AR technology synchronously superimposes virtual information and real environment, and people can experience the virtual-real combined scene by using AR devices. AR scene includes entity scene and virtual model, and generally through converting and spatially calibrating different coordinate systems of entity scene and virtual model, the virtual model and the entity scene are fused and displayed.

[0003] Since the AR module is generally run on a mobile terminal, the acquisition of coordinate position usually depends on the GPS positioning module, and the positioning accuracy is poor. At the same time, the direction angle obtained by relying on the gyroscope sensor also usually has errors. The more mainstream ARCore, ARkit and other ARSDKs on the market also have the problem of increasing position and direction error accumulation after running for a period of time.

[0004] In view of the above problems, the current solution often focuses on improving the accuracy of GNSS equipment and angle measuring instruments, manually adjusting, or collecting feature point clouds in advance. When collecting, the position and angle of the model relative to a single feature point are bound. Next time when entering the same position, the feature point cloud recognized is relied on for display. However, even if high-precision GNSS equipment is relied on, there are factors such as weather interference and building obstruction, which affect the accuracy or make it impossible to obtain high-precision coordinates. And relying on the way of collecting point clouds in advance, if the feature points are not updated in time, the recognition of the feature points will become extremely difficult, and collecting feature points in all areas regularly or irregularly will require a lot of work, which is less practical. SUMMARY

[0005] In view of the problems of poor anti-interference ability of traditional methods, large feature point collection workload and lack of continuous update of feature points, the present application provides an AR model pose display continuous correction method, device and storage device. The technical scheme of the method is as follows:

[0006] S1, connect the AR device with the GNSS device, initialize the device and judge the GNSS device state;

[0007] S2, start the camera, identify and extract feature points by ORB and obtain the AR coordinates of the feature points;

[0008] S3, calculate the average conversion relationship matrix of the plane geographic coordinate system and the AR coordinate system;

[0009] S4, calculating the pose of the object in the AR scene according to the average conversion relationship matrix, and adjusting the AR model in the picture;

[0010] S5, calculating the plane geographic coordinates of the feature points when the device is normal, and updating the feature information and the geographic coordinates of the feature points.

[0011] Further, the high-precision GNSS device is connected with the AR device in a Bluetooth manner, and the GNSS working state is divided into normal and abnormal; the plane geographic coordinates and the AR coordinates are obtained from the GNSS device and the AR device in the normal state; the user is prompted that the GNSS device is not working normally in the abnormal state, and the plane geographic coordinates of the feature points are obtained from the storage; the GNSS device is determined to be abnormal when at least one of the following conditions is met: the error exceeds the threshold value within a certain time, the coordinates cannot be obtained, and the delay of obtaining the coordinates exceeds the time limit.

[0012] Further, the feature points are points where the image gray value changes sharply or target object points with large curvature on the image edge, which effectively reflect the essential characteristics of the image, and the ORB algorithm (OrientedFAST and Rotated BRIEF) provided by OpenCV is used to identify and extract the feature points in the picture; the AR coordinates are position data in the AR coordinate system; the AR coordinate system is provided by ARSDK, and the position of the camera of the AR device at the time of ARSDK initialization is taken as the origin, and the XYZ axes are defined as right, up and forward directions of the device.

[0013] Further, the specific steps of the ORB algorithm identification and extraction are as follows:

[0014] S21, using the FAST (Features from Accelerated Segment Test) algorithm to find the key points in the image;

[0015] S22, using the BRIEF (Binary Robust Independent Elementary Features) algorithm to generate image feature descriptors;

[0016] S23, improving the image feature descriptors based on the rotation invariance and scale invariance algorithm.

[0017] Further, when the GNSS device is abnormal, the average conversion relationship matrix between the plane geographic coordinate system and the AR coordinate system is calculated by selecting the feature points; when the GNSS device is normal, the average conversion relationship matrix between the plane geographic coordinate system and the AR coordinate system is calculated by using the GNSS device coordinates.

[0018] Further, the average conversion relationship matrix is calculated by the following formula:

[0019] m=m1*R*m2

[0020] Wherein,

[0021]

[0022]

[0023]

[0024] In the formula, A x1 and A x2 are the X-axis coordinates of the AR coordinate system of the two selected feature points, P x1 and x2 are the X-axis coordinates of the plane geographic coordinate system, and θ is the rotation angle between the AR coordinate system and the plane geographic coordinate system.

[0025] Further, the specific steps of adjusting the pose of the AR model in the picture when the object is matched and displayed in the AR scene according to the average conversion relationship matrix are as follows:

[0026] S41, calculate the coordinates of the AR model in the AR coordinate system;

[0027] S42, assign the coordinates and rotation matrix of the AR model to the display coordinates in the ARSDK;

[0028] S43, update the pose according to the display coordinates and redraw the AR model.

[0029] Further, the specific steps of updating the feature information and geographic coordinates of the feature points when the device is normal are as follows:

[0030] S51, calculate the current plane geographic coordinates of the updated feature points using the average conversion relationship matrix between the AR coordinate system and the plane geographic coordinate system;

[0031] S52, calculate the average plane geographic coordinate result of the feature points based on the conversion relationship matrix;

[0032] S53, update the historical plane geographic coordinates and feature information of the feature points with the average plane geographic coordinate result and store them.

[0033] A storage device stores instructions and data for implementing an AR model pose display continuous correction method.

[0034] The AR model pose display continuous correction device comprises a processor and a storage device; the processor loads and executes instructions and data in the storage device to realize an AR model pose display continuous correction method. BRIEF DESCRIPTION OF DRAWINGS

[0035] The application will be further described below in combination with the drawings and examples, wherein:

[0036] Figure 1 is a flowchart of an AR model pose display continuous correction method in an embodiment of the application;

[0037] Figure 2 is a schematic diagram of a conversion relationship matrix calculation process of a plane geographic coordinate system and an AR coordinate system in an embodiment of the application;

[0038] Figure 3 is a flowchart of AR model pose display and adjustment of an in-plane AR model in an embodiment of the application;

[0039] Figure 4 is a flowchart of updating feature information and geographic coordinates of a stored feature point in an embodiment of the application;

[0040] Figure 5 is a schematic diagram of hardware device work in an embodiment of the application. DETAILED DESCRIPTION

[0041] In order to have a clearer understanding of the technical features, objectives and effects of the application, the specific embodiments of the application will be described in detail with reference to the drawings.

[0042] An embodiment of the application provides an AR model pose display continuous correction device, which comprises a processor and a storage device and is used to realize the AR model pose display continuous correction method.

[0043] Please refer to Figure 1 , Figure 1 is a flowchart of an AR model pose display continuous correction method in an embodiment of the application, and specifically comprises the following steps:

[0044] Firstly, connect the AR device and the GNSS device, initialize the devices and judge the GNSS device state.

[0045] Connect the AR device and the high-precision GNSS device in Bluetooth mode; in the normal state, obtain the plane geographic coordinates and the AR coordinates from the GNSS device and the AR device; in the abnormal state, prompt the user that the GNSS device is not working properly, and obtain the plane geographic coordinates of the feature points from the storage; the way of determining the GNSS device abnormality is at least one of the following: the error is greater than E within the duration T; the coordinates cannot be obtained; the delay of obtaining the coordinates is greater than t; in this embodiment, T is 5 seconds, E is 0.5 meters, and t is 2 seconds.

[0046] Second step, start the camera, identify and extract the feature points by using ORB, and obtain the AR coordinates of the feature points.

[0047] Third step, calculate the conversion relationship matrix of the plane geographic coordinate system and the AR coordinate system.

[0048] When the GNSS device is abnormal, select the feature points to calculate the average conversion relationship matrix of the plane geographic coordinate system and the AR coordinate system; when the GNSS device is normal, use the GNSS device coordinates to calculate the average conversion relationship matrix of the plane geographic coordinate system and the AR coordinate system; calculate the average conversion relationship matrix by using the average method or the least square method multiple times; when selecting, pay attention to the fact that the distance between the two feature points is not less than 2 meters, and the staying time is not less than 1 second, and select a long distance to ensure the accuracy of the coordinate acquisition.

[0049] Fourth step, calculate the pose of the object in the AR scene when matching and displaying, and adjust the AR model in the picture according to the average conversion relationship matrix; assign the calculated coordinates Ax and the rotation matrix R to the display coordinates in the ARSDK, update the pose, and redraw the AR model.

[0050] Fifth step, when the device is normal, calculate the plane geographic coordinates of the recognized feature points, and update the feature information and the geographic coordinates of the stored feature points.

[0051] Please refer to Figure 2 , Figure 2 The conversion relationship matrix of the plane geographic coordinate system and the AR coordinate system is calculated in the embodiment of the application.

[0052] First step, define two selected feature points as point 1 and point 2, and the plane geographic system coordinates of the two points are P1(P x1 ,0,P z1 ) and P2(P x2 ,0,P z2 ), respectively, and the AR coordinate system coordinates are A1(A x1 ,0,A z1 ) and A2(A x2 ,0,A z2 ).

[0053] Second step, establish the inertial coordinate system O1 with A1 as the origin, the direction and unit length of which are the same as AR coordinate system, that is, unit matrix translation A1 coordinate, the transformation relationship between the coordinate system and AR coordinate system is expressed by matrix as follows:

[0054]

[0055] Third step, the rotation angle between AR coordinate system and plane geographic coordinate system is the angle between vectors v1(A2-A1) and v2(P2-P1), which is expressed by rotation matrix R as follows:

[0056]

[0057] In the formula,

[0058] Fourth step, establish the inertial coordinate system O2 with P1 as the origin, the direction and unit length of which are the same as plane geographic coordinate system, that is, unit matrix translation-P1 coordinate, the transformation relationship between the coordinate system and plane geographic coordinate system is expressed by matrix as follows:

[0059]

[0060] Fifth step, the transformation relationship between AR coordinate system and plane geographic coordinate system is expressed by matrix as follows:

[0061] m=m1*R*m2

[0062] Sixth step, multiple selection is performed for calculation, and average transformation relationship matrix is obtained by using average method or least square method

[0063] Please refer to Figure 3 , Figure 3 The specific steps for displaying and adjusting the in-plane AR model in the AR model pose display and adjustment in the embodiment of the application are as follows:

[0064] First step, calculate the coordinates of the AR model in the AR coordinate system, that is, the coordinates A x The calculation formula is as follows:

[0065]

[0066] In the formula, P x is the real plane geographic coordinates of the AR model.

[0067] Second step, assign the AR model coordinates A x and the rotation matrix R to the display coordinates in ARSDK.

[0068] Third step, update the pose of the AR model according to the display coordinates, and redraw the AR model.

[0069] Please refer to Figure 4 , Figure 4 The specific steps for updating the feature information and geographic coordinates of the feature points are as follows:

[0070] First, the average conversion relationship matrix between the AR coordinate system and the plane geographic coordinate system is used The current plane geographic coordinates of the updated feature points are calculated, and the plane geographic coordinates P y of the feature points are calculated as follows:

[0071]

[0072] In the formula, is the inverse matrix of .

[0073] Second, the results Py1, Py2, …, Pyk calculated based on the k times conversion relationship matrix are used to calculate the average plane geographic coordinate result

[0074] Third, the historical plane geographic coordinates and feature information of the feature points are updated with the average plane geographic coordinate result and stored.

[0075] Please refer to Figure 5 , Figure 5 is a hardware device working schematic diagram of an embodiment of the application, which specifically includes: an AR model pose display continuous correction device 401, a processor 402, and a storage device 403.

[0076] An AR model pose display continuous correction device 401: the AR model pose display continuous correction device 401 realizes the AR model pose display continuous correction method.

[0077] Processor 402: the processor 402 loads and executes instructions and data in the storage device 403 to realize the AR model pose display continuous correction method.

[0078] Storage device 403: the storage device 403 stores instructions and data; the storage device 403 is used to realize the AR model pose display continuous correction method.

[0079] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not limited thereto; although the present application has been described in detail with reference to the foregoing examples, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

[0080] The beneficial effects of the present application are: to realize the display and continuous correction of the AR model pose, and to ensure the accuracy of the matching of the AR scene and the reality. Meanwhile, considering the abnormal situation of high-precision GNSS equipment, the position of the feature points is stored to calculate the high-precision pose of the AR equipment, and to ensure the matching of the AR scene and the reality scene. And the mean value of the multiple plane geographic coordinates is calculated to improve the accuracy of the feature point plane geographic coordinates.

[0081] The above is only a preferred embodiment of the present application, and is not intended to limit the present application, and any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for continuous correction of pose display of an AR model, characterized in that, Includes the following steps: S1. Connect the AR device and the high-precision GNSS device, initialize the device and determine the status of the GNSS device; S2. Start the camera, use the ORB algorithm to identify and extract feature points and obtain the AR coordinates of the feature points; S3. Calculate the average transformation matrix between the planar geographic coordinate system and the AR coordinate system. The average transformation matrix is ​​obtained by multiple calculations using the average method or the least squares method. S4. Calculate the pose of the object when it is matched and displayed in the AR scene according to the average transformation relationship matrix, and adjust the AR model in the picture; the specific steps of calculating the pose of the object when it is matched and displayed in the AR scene and adjusting the AR model in the picture are as follows: S41. Calculate the coordinates of the AR model in the AR coordinate system; S42. Assign the AR coordinates and rotation matrix of the model to the display coordinates in ARSDK; S43. Update the pose using the display coordinates and redraw the AR model. S5. When the equipment is working properly, calculate the planar geographic coordinates of the feature points and update the stored feature information and geographic coordinates of the feature points. Step S5 is as follows: S51. Calculate the updated planar geographic coordinates of the feature points using the average transformation matrix between the AR coordinate system and the planar geographic coordinate system. S52. Calculate the average planar geographic coordinates of feature points based on the planar geographic coordinates calculated using the transformation relation matrix; S53. Update the historical planar geographic coordinates and feature information of the feature point with the average planar geographic coordinate results and store them.

2. The AR model pose display continuous correction method as described in claim 1, characterized in that: The high-precision GNSS device and the AR device are connected via Bluetooth; the GNSS device has two operating states: normal and abnormal. In normal operation, planar geographic coordinates and AR coordinates are obtained from GNSS and AR devices; in abnormal operation, the user is prompted that the GNSS device is not working properly, and the planar geographic coordinates of feature points are obtained from storage; the method for determining GNSS device abnormality is to meet at least one of the following: the error exceeds a threshold within a certain period of time; coordinates cannot be obtained; the delay in obtaining coordinates exceeds the time limit.

3. The AR model pose display continuous correction method as described in claim 2, characterized in that: The feature points include points in the image where the grayscale value changes drastically, reflecting the essential features of the image, and target object points with large curvature on the edge of the image; the AR coordinates are position data in the AR coordinate system; the AR coordinate system is provided by ARSDK, with the location of the AR device's camera at the time of ARSDK initialization as the origin, and the coordinate axes defined by the orientation.

4. The AR model pose display continuous correction method as described in claim 3, characterized in that... The specific steps for the ORB algorithm to identify and extract feature points are as follows: S21. Use the FAST algorithm to find key points in the image; S22. Use the BRIEF algorithm to generate image feature descriptors; S23. Image feature descriptor algorithm improvement based on rotation invariance and scale invariance.

5. The AR model pose display continuous correction method as described in claim 4, characterized in that: When the GNSS equipment malfunctions, select feature points to calculate the average transformation matrix between the planar geographic coordinate system and the AR coordinate system; when the GNSS equipment is functioning normally, use the GNSS equipment coordinates to calculate the average transformation matrix between the planar geographic coordinate system and the AR coordinate system.

6. The method for continuous correction of AR model pose display as described in claim 5, characterized in that, The formula for calculating the average transformation relation matrix is: m=m1*R*m2 in, In the formula, A x1 and A x2 The X-axis coordinates of the two selected feature points in the AR coordinate system are given by P. x1和 P x2 Let θ be the X-axis coordinate of its planar geographic coordinate system, and θ be the rotation angle between the AR coordinate system and the planar geographic coordinate system.

7. A storage device that stores instructions and data for implementing a continuous correction method for AR model pose display as described in any one of claims 1 to 6.

8. An AR model pose display continuous correction device, comprising: The processor and the storage device; The processor loads and executes instructions and data in the storage device to implement a continuous correction method for AR model pose display as described in any one of claims 1 to 6.

Images