An augmented reality tracking registration method, system, and medium

Abstract

The application relates to an augmented reality tracking registration method, a system and a medium, and the augmented reality tracking registration method comprises the following steps: collecting binocular images of an industrial site scene in real time by a mobile binocular camera, and constructing a transformation matrix of an artificial mark coordinate system corresponding to a tracking registration result based on the artificial mark; point and line features in the binocular images of the industrial site scene are extracted and matched frame by frame respectively, point feature screening and vanishing point line feature screening are performed, a global pose graph is obtained based on a screened point and line feature set, and tracking positioning of a target AR virtual object is performed; a re-projection error model of point and line feature fusion is constructed, and an optimized global pose graph is obtained; the real-time pose of the binocular camera is estimated in real time, and the target AR virtual object is registered in the artificial mark coordinate system based on the real-time pose. The application can provide more accurate, robust and more industrial environment adaptable augmented reality for industrial augmented reality application in a complex industrial environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of augmented reality tracking registration in industrial scenarios. In particular, it relates to an augmented reality tracking registration method, system and medium. BACKGROUND

[0002] The in-depth application of augmented reality (AR) technology in the industrial field, especially in key links such as assembly work and design verification, not only improves the efficiency and quality of industrial production, but also promotes the intelligent and fine development of industrial production. In this process, tracking registration technology, as the core of AR application, directly determines the user experience and system reliability. With the development of sensors and information technology, the existing multi-sensor fusion tracking registration method can obtain relatively accurate pose estimation results to provide strong support for AR application. However, most current tracking registration methods focus on general scenarios in design, and do not fully consider the challenges unique to industrial environments, such as extreme lighting conditions, non-ideal surface materials, and texture loss. These factors greatly limit the application effect of traditional tracking registration methods based on single-point features in industrial sites, increasing the risk of tracking loss and registration failure.

[0003] In view of this limitation of AR application in industrial scenarios, there is an urgent need for a tracking registration method more suitable for dynamic industrial scenarios, aiming to effectively deal with interference factors in dynamic industrial scenarios, thereby providing more accurate tracking registration results for AR application in industrial scenarios to support the development of industrial automation and intelligentization. SUMMARY

[0004] The present application provides an augmented reality tracking registration method, system and medium. To solve the problem that traditional tracking registration methods based on single-point features increase the risk of tracking loss and registration failure in industrial environments in industrial sites.

[0005] To achieve the above-mentioned purpose, in a first aspect, the present application provides an augmented reality tracking registration method for industrial sites with a movable binocular camera and a mobile hardware display module, comprising:

[0006] The movable binocular camera is used to collect industrial site scene binocular images in real time, and the world coordinate system, AR virtual model coordinate system and mobile hardware platform display module coordinate system are initialized and aligned based on artificial markers to construct a transformation matrix of the tracking registration result corresponding to the artificial marker coordinate system;

[0007] extract and match point line features in the industrial scene binocular images frame by frame respectively, perform screening of point features in the point line features and line feature screening based on vanishing points of the industrial scene binocular images, obtain a global pose graph based on a screened point line feature set, and perform tracking and positioning of a target AR virtual object;

[0008] construct a re-projection error model of the point line feature fusion, obtain a re-projection error based on the re-projection error model, and obtain an optimized global pose graph based on the re-projection error;

[0009] estimate a real-time pose of the binocular camera in real time based on the optimized global pose graph, and register a target AR virtual object in the artificial mark coordinate system based on the real-time pose.

[0010] To achieve the above object, in a second aspect, the application further relates to an augmented reality tracking and registration system for an industrial scene with a movable binocular camera and a mobile hardware display module, comprising:

[0011] a coordinate transformation unit, configured to collect industrial scene binocular images in real time by moving the binocular camera, and complete initialization and alignment of the world coordinate system, an AR virtual model coordinate system and a mobile hardware platform display module coordinate system based on artificial marks to construct a transformation matrix of a tracking and registration result corresponding to an artificial mark coordinate system;

[0012] a feature screening unit, configured to extract and match point line features in the industrial scene binocular images frame by frame respectively, perform screening of point features in the point line features and line feature screening based on vanishing points of the industrial scene binocular images, obtain a global pose graph based on a screened point line feature set, and perform tracking and positioning of a target AR virtual object;

[0013] a global pose graph optimization unit, configured to construct a re-projection error model of the point line feature fusion, obtain a re-projection error based on the re-projection error model, and obtain an optimized global pose graph based on the re-projection error;

[0014] a target registration unit, configured to estimate a real-time pose of the binocular camera in real time based on the optimized global pose graph, and register a target AR virtual object in the artificial mark coordinate system based on the real-time pose.

[0015] Preferably, the binocular camera is provided with a sensor module, and the sensor module data comprises binocular fisheye image frames and accelerometer and gyroscope data.

[0016] The coordinate transformation unit is specifically configured to:

[0017] aligning the world coordinate system through the artificial marker;

[0018] Suppose the conversion matrix of the AR virtual model coordinate system to the artificial marker coordinate system is v T m The process of the initialization alignment is equivalent to the process of obtaining the pose of the AR virtual model relative to the mobile hardware platform display module, and the conversion matrix is v T d Description, meet:

[0019] v T d = v T m · m T d · d T t · pose T t · d T t -1

[0020] Wherein, m T d is the conversion matrix between the artificial marker coordinate system and the mobile hardware platform display module coordinate system, d T t is the conversion matrix between the sensor module data and the mobile hardware platform display module, and the real-time pose estimation during movement is represented as pose T t The conversion matrix v T d is used to establish subsequent tracking registration results under the artificial marker coordinate system.

[0021] To achieve the above object, the third aspect, the application also relates to a computer readable storage medium, the storage medium stores instructions, the instructions run to execute the above-mentioned augmented reality tracking registration method.

[0022] The augmented reality tracking registration method, system and medium provided by the application have the following beneficial effects compared with the prior art:

[0023] The application aims to provide an industrial augmented reality tracking registration method, system and medium based on point-line feature fusion, which effectively describes the shape and structure of objects in an industrial scene by fusing point-line features in the environment, provides accurate information for subsequent matching and positioning, and further improves the adaptability of the augmented reality tracking registration method to the industrial scene. The application can provide a more accurate, robust, and more adaptable augmented reality tracking registration scheme for industrial augmented reality applications in complex industrial environments, and can achieve accurate virtual-real fusion effects in industrial sites. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 Method flow of the augmented reality tracking registration method in the embodiment one of the application Figure 1 ;

[0025] Figure 2 Method flow of the augmented reality tracking registration method in the embodiment one of the application Figure 2 ;

[0026] Figure 3 Application system block diagram of the augmented reality tracking registration method in the embodiment one of the application

[0027] Figure 4 Structure block diagram of the augmented reality tracking registration system in the embodiment one of the application DETAILED DESCRIPTION

[0028] The application will be further described in detail below in combination with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the application, but not to limit the application. In addition, it should be noted that, for the convenience of description, only the parts related to the application are shown in the drawings, not all the structures.

[0029] Embodiment one

[0030] An augmented reality tracking registration method, please refer to Figures 1-3 , which is used in an industrial site with a movable binocular camera and a mobile hardware display module, and includes S10-S40.

[0031] S10: The movable binocular camera collects industrial site scene binocular images in real time, and completes the initialization alignment of the world coordinate system, the AR virtual model coordinate system and the mobile hardware platform display module coordinate system based on artificial markers, to construct a transformation matrix of the tracking registration result corresponding to the artificial marker coordinate system.

[0032] Preferably, the binocular camera is provided with a sensor module, and the sensor module data includes binocular fisheye image frames and accelerometer and gyroscope data, and the transformation matrix of the tracking registration result corresponding to the artificial marker coordinate system is constructed, specifically including S11-S13:

[0033] S11: Real-time collection of binocular images of industrial scene by mobile binocular camera.

[0034] Specifically, the sensor module data of the real-time industrial scene is collected, including binocular images of the industrial scene and accelerometer and gyroscope data.

[0035] S12: Alignment of the world coordinate system through artificial markers.

[0036] Specifically, the artificial markers for system initialization and world coordinate system alignment are used. The initial pose coordinate system of system initialization is established with the camera optical center as the origin.

[0037] S13: The conversion matrix of the AR virtual model coordinate system to the artificial marker coordinate system is v T m The initialization alignment process is equivalent to obtaining the pose of the AR virtual model relative to the mobile hardware platform display module. The transformation matrix v T d is described, which satisfies:

[0038] v T d = v T m · m T d · d T t · pose T t · d T t -1

[0039] wherein, m T d is the conversion matrix between the artificial marker coordinate system and the mobile hardware platform display module coordinate system, d T t is the conversion matrix between the sensor module data and the mobile hardware platform display module, and the real-time pose estimation during movement is represented as pose T t The subsequent tracking registration results are established in the artificial marker coordinate system through the transformation matrix v T d

[0040] S20: Point and line features in the binocular images of the industrial scene are extracted and matched frame by frame, respectively. Point features in the point and line features are screened, and line features based on the vanishing points of the binocular images of the industrial scene are screened. Based on the screened point and line feature sets, a global pose graph is obtained, and the target AR virtual object is tracked and positioned. ​

[0041] Specifically, as shown in Figure 2 S20 includes S21-S24.

[0042] S21: For the collected frame-by-frame industrial scene binocular image, extract the ORB (Oriented FAST and Rotated BRIEF) point feature and the LSD (Line Segment Detector) line feature, and obtain the LBD (Line Segment Descriptor) descriptor of the extracted line feature.

[0043] S22: Match the point and line features of the industrial scene binocular image respectively, and preliminarily screen the line features based on the relative line segment length of the extracted line features, wherein the preliminary screening includes: sorting the extracted line segments by the relative line segment length; screening and removing the line segments whose sorting exceeds a first preset value, to obtain a line segment sequence L' after preliminary screening, wherein the first preset value represents the maximum value of the relative line segment length of the extracted line segments.

[0044] In this embodiment, the relative line segment length is represented as:

[0045]

[0046] Wherein, len i represents the relative length value of the i-th line segment extracted from the image frame, and (x sl ,y sl ) represents the starting coordinates of the line segment, (x el ,y el ) represents the end coordinates of the line segment, W i is the width of the image, H i is the length of the image, and η is the scaling coefficient of the image; all the line segments len i values extracted from the image frame are sorted in reverse order.

[0047] S23: Calculate the image vanishing point based on the line segment sequence L' after preliminary screening.

[0048] The calculation of the image vanishing point includes S231-S233.

[0049] S231: Construct an equivalent spherical polar coordinate grid of the industrial scene binocular image to convert the pixel (x, y) to (φ, λ) on the equivalent spherical coordinate system.

[0050] In this embodiment, specifically, the equivalent spherical polar coordinate grid of the image is constructed as follows:

[0051] Wherein, taking the image plane parallel direction as the equivalent unit sphere X axis and Y axis direction, the Z axis points to the image plane optical center, the optical center (x0, y0) and the focal length f are given, the equivalent unit sphere longitude and latitude are represented as (φ, λ);

[0052] S232: Vanishing point hypothesis generation: iteratively randomly select two line segments and calculate the intersection point, and project the intersection point to the unit vector on the equivalent spherical surface to obtain the first vanishing point v1; assuming the second vanishing point v2 = (X2, Y2, Z2), traverse the assumption of the second vanishing point in the equivalent spherical polar coordinate grid, and calculate the second vanishing point through the orthogonal constraint; the third vanishing point v3 is calculated through v3 = v1 x v2.

[0053] Wherein, the orthogonal constraint specifically includes: Wherein the first vanishing point v1 = (X1, Y1, Z1).

[0054] S233: Best vanishing point estimation: for the line segment sequence L' after preliminary screening, the intersection point p and its corresponding spherical grid between the two line segments are calculated, and the intersection point p of the spherical grid i is represented as α (λ i ,φ i ) with 1 interval uniform sampling, and taken as the grid response value, for each vanishing point hypothesis, the maximum response value hypothesis is obtained by calculating the vanishing point line segment combination response value. Take as the best vanishing point estimation.

[0055] S24: Cluster all extracted line features with vanishing points, and perform line feature screening and rejection based on the clustering of the extracted line feature vanishing points.

[0056] Specifically, the clustering of the vanishing point is classified by the angle deviation of the line feature in the line segment sequence L' after preliminary screening and the first vanishing point, the second vanishing point and the third vanishing point, wherein the angle deviation is obtained by calculating the angle θ between the straight line l mid obtained by connecting the line segment midpoint and the vanishing point, when the angle θ satisfies less than the second preset value, it is considered that the line segment is classified as the vanishing point, and three classification sequences of line features are obtained on the basis of the line segment sequence L' after clustering. The line segment that satisfies the condition is filtered to obtain the line feature sequence L'' for tracking.

[0057] S30: Construct a re-projection error model of point-line feature fusion, and obtain a re-projection error based on the re-projection error model, and obtain an optimized global pose graph based on the re-projection error.

[0058] In this embodiment, specifically, for the re-projection error of the line feature, p l , ql denote the three-dimensional projection of the detected two-dimensional line segment endpoints, so the normalized line coefficients I satisfy:

[0059]

[0060] denote the estimated line coefficients of the observed endpoints (p i,j i,j ) of line segment j in the i-th frame, and K -1 denote the process of projecting a two-dimensional point into a three-dimensional point through intrinsic parameters, define the re-projection error of the line feature as the error of the endpoints to the observed line, denoted as:

[0061]

[0062] Based on the collected sensor data, feature points are detected and described for each frame of binocular image, and the feature points in the binocular image are matched, the pose is estimated through the IMU (Inertial Measurement Unit) data, and the camera pose is calculated using the adjacent frame relationship, denoted as S i denote the state variable containing the pose T i , the velocity v i and the gyroscope and accelerometer bias , denote the covariance matrix of the pre-integrated rotation, velocity and position measurement vectors, denoted as denote the inertial residual, r ij denote the point feature re-projection error between the i-th frame and the point j at the pose x j , r ij l denote the re-projection error of the line feature, given a set of k+1 key frames and state The back-end optimization problem of fusing visual inertial is represented as the combination of IMU residual and point-line re-projection error, that is,

[0063]

[0064] K j is the key frame set that observes the three-dimensional point and the three-dimensional line, the result of the pose estimation is optimized through the above nonlinear optimization process, and the pose and position of the binocular camera are calculated according to the matching result of the feature points between the current frame binocular image and the previously stored frame binocular image in the map, so as to obtain the optimized global pose graph.

[0065] S40: Real-time estimate the real-time pose of the binocular camera based on the optimized global pose graph, and register the target AR virtual object in the artificial marker coordinate system based on the real-time pose.

[0066] ​​In this embodiment, based on the real-time pose, unity is used to draw and import virtual objects in the artificial marker coordinate system to register the target AR virtual object.

[0067] After registering the target AR virtual object, the mobile hardware platform display module presents augmented reality based on the registered target AR virtual object with the real industrial scene binocular image collected in real time as the real object part.

[0068] In an example of this embodiment, the verification environment used is an Intel i3-10100Y processor with a 1.61GHz main frequency and 8G memory, and the software is Visual Studio 2019 version and Unity 2021.3.16 version.

[0069] The augmented reality tracking registration method of this embodiment first acquires sensor data of the industrial site environment, including real scene image frames and inertial data such as accelerometer and gyroscope data, and initializes alignment through artificial markers. Secondly, pose estimation and tracking registration are performed through the industrial augmented reality tracking registration method based on point-line feature fusion. Finally, augmented reality picture rendering is performed based on the tracking registration result, and the effect is presented in the mobile hardware platform reality module. Therefore, the industrial augmented reality tracking registration system based on point-line feature fusion can be realized, and the problem of insufficient accuracy and robustness of the augmented reality tracking registration method in the industrial environment can be solved.

[0070] Embodiment two

[0071] As shown in Figure 4 An augmented reality tracking registration system for an industrial site with a movable binocular camera and a mobile hardware display module. The augmented reality tracking registration system of this embodiment can be implemented in electronic devices with central processing units, such as personal computers, upper computers, servers, and smart terminals. In this embodiment, the augmented reality tracking registration system includes a coordinate transformation unit 61, a feature selection unit 62, a global pose graph optimization unit 63, and a target registration unit 64.

[0072] The coordinate transformation unit 61 is used to collect industrial site scene binocular images in real time with a mobile binocular camera, and to complete the initialization alignment of the world coordinate system, AR virtual model coordinate system, and mobile hardware platform display module coordinate system based on artificial markers to construct a transformation matrix of the tracking registration result corresponding to the artificial marker coordinate system.

[0073] The feature screening unit 62 is configured to extract and match point line features in the binocular image of the industrial scene respectively frame by frame, screen the point features in the point line features, and screen the line features based on the vanishing points of the binocular image of the industrial scene, obtain a global pose graph based on the screened point line feature set, and track and position the target AR virtual object.

[0074] The global pose graph optimization unit 63 is configured to construct a re-projection error model of the point line feature fusion, obtain a re-projection error based on the re-projection error model, and obtain an optimized global pose graph based on the re-projection error.

[0075] The target registration unit 64 is configured to estimate a real-time pose of the binocular camera based on the optimized global pose graph, and register the target AR virtual object in the artificial marker coordinate system based on the real-time pose.

[0076] After the target AR virtual object is registered, the mobile hardware platform display module presents augmented reality based on the registered target AR virtual object and the binocular image of the industrial scene collected in real time.

[0077] In some embodiments, the binocular camera is provided with a sensor module.

[0078] The coordinate transformation unit 61 is specifically configured to:

[0079] Collect sensor module data of a real-time industrial scene, wherein the sensor module data includes binocular images of the industrial scene and accelerometer and gyroscope data.

[0080] Align the world coordinate system through the artificial marker;

[0081] Suppose the conversion matrix of the AR virtual model coordinate system to the artificial marker coordinate system is v T m The process of initializing the alignment is equivalent to the process of obtaining the pose of the AR virtual model relative to the mobile hardware platform display module, and the transformation matrix v T d is used to describe, and satisfies:

[0082] v T d = v T m · m T d · d T t · pose T t · d T t -1

[0083] Wherein, mT d is a conversion matrix between the artificial marker coordinate system and the mobile hardware platform display module coordinate system, d T t is a conversion matrix between the sensor module data and the mobile hardware platform display module, and the real-time pose estimation in the mobile process is represented as pose T t , and the subsequent tracking registration result is established in the artificial marker coordinate system through the conversion matrix v T d .

[0084] The implementation process, method and effect of the augmented reality tracking registration system of the embodiment are the same as those of the augmented reality tracking registration method described in the first embodiment, and thus will not be repeated here.

[0085] Embodiment Three

[0086] The present application relates to a computer-readable storage medium, and the storage medium stores instructions, and the instructions execute the augmented reality tracking registration method of the first embodiment when running, and the implementation process, method and effect thereof are the same as those of the augmented reality tracking registration method described in the first embodiment, and thus will not be repeated here.

[0087] It should be noted that, in this document, the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such a process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0088] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation using the content of the present application specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An augmented reality tracking registration method, characterized in that: For industrial sites with movable binocular cameras and mobile hardware display modules, including: The binocular camera is moved to collect binocular images of the industrial scene in real time, and initialization alignment of the world coordinate system, the AR virtual model coordinate system, and the coordinate system of the mobile hardware platform display module is completed based on the artificial marker, so as to construct a transformation matrix corresponding to the artificial marker coordinate system of the tracking registration result; Extracting and matching point and line features in the binocular image of the industrial scene frame by frame, filtering point features among the point and line features, and filtering line features based on the vanishing points of the binocular image of the industrial scene, obtaining a global pose graph based on the filtered point and line feature set, and tracking and positioning the target AR virtual object; Constructing a reprojection error model of the point-line feature fusion, obtaining a reprojection error based on the reprojection error model, and obtaining the optimized global pose graph based on the reprojection error; estimating the real-time pose of the binocular camera in real time based on the optimized global pose graph, and registering the target AR virtual object in the artificial mark coordinate system based on the real-time pose; The binocular camera is provided with a sensor module, and the moving the binocular camera to collect binocular images of the industrial scene in real time includes: collecting sensor module data of the real-time industrial scene, wherein the sensor module data includes the binocular image of the industrial scene and accelerometer and gyroscope data; The initialization alignment of the world coordinate system, the virtual model coordinate system, and the coordinate system of the mobile hardware platform display module based on the artificial marker is completed to construct a transformation matrix corresponding to the artificial marker coordinate system of the tracking registration result, specifically including: Aligning the world coordinate system by artificial markers; Assume that the transformation matrix from the AR virtual model coordinate system to the artificial marker coordinate system is , the initialization alignment process is equivalent to the process of obtaining the posture of the AR virtual model relative to the display module of the mobile hardware platform, using the transformation matrix Description, meet: ； in, is the transformation matrix between the coordinate system of the artificial marker and the coordinate system of the mobile hardware platform display module, is the conversion matrix between the sensor module data and the mobile hardware platform display module, and the real-time pose estimation during the movement is expressed as , through the transformation matrix To establish the subsequent tracking registration results in the artificial identification coordinate system; The extracting and matching the point and line features in the binocular image of the industrial scene frame by frame, screening the point features in the point and line features, and screening the line features in the point and line features based on the vanishing points of the industrial scene image, and tracking and positioning based on the screened point and line feature set, include: Extracting ORB point features and LSD line features from the collected frame-by-frame binocular images of the industrial scene, and obtaining LBD descriptors of the extracted line features; Match the point and line features of the binocular image of the industrial scene respectively, and perform preliminary screening of the line features based on the relative line segment lengths of the extracted line features, wherein the preliminary screening includes: sorting the extracted line segments by relative line segment lengths; screening and eliminating line segments whose sorting exceeds a first preset value, and obtaining a line segment sequence after preliminary screening , wherein the first preset value represents the maximum relative line segment length among the extracted line segments; Based on the segment sequence after the preliminary screening Calculate the image vanishing point, including: Construct an equivalent spherical polar coordinate grid of the binocular image of the industrial scene to divide the pixels Transformed to the equivalent spherical coordinate system : Vanishing point hypothesis generation: Iterate two randomly selected line segments and calculate the intersection point, and project the intersection point to the unit vector on the equivalent sphere to obtain the first vanishing point v 1; Assume the second vanishing point , traverse the assumption that the second vanishing point is in the equivalent spherical polar coordinate grid, and calculate the second vanishing point through the orthogonal constraint; Calculate the third vanishing point v 3; Best vanishing point estimation: For the line segment sequence after the preliminary screening , calculate the intersection points between each pair of line segments p and its corresponding spherical grid, Indicates A spherical grid with uniformly spaced samples i intersection p The number of vanishing point segments is used as the grid response value. For each vanishing point hypothesis, the maximum response value hypothesis is obtained by calculating the combined response value of the vanishing point segments. As the best vanishing point estimate; The line feature screening includes: clustering the vanishing point as the center to form a line segment sequence that has passed the preliminary screening The angle deviations between the line features in the first vanishing point, the second vanishing point and the third vanishing point are classified, wherein the angle deviations are calculated by calculating the line segment sequence The straight line connecting the midpoint of the midline segment and the vanishing point With the line segment sequence Angle between midline segments Get, when the angle If the value is less than the second preset value, the line segment is considered to be classified as the vanishing point. The result is clustered in the line segment sequence. Three classification sequences of line features are obtained based on , for conditions that meet Filter the line segments to obtain the line feature sequence for tracking .

2. The augmented reality tracking registration method according to claim 1, characterized in that: The sorting of the extracted line segments by relative line segment lengths includes: The relative line segment length is expressed as: (2) in, Indicates the first i The relative length of the line segment, Indicates the starting coordinates of the line segment, represents the end coordinates of the line segment, is the width of the image, is the length of the image, The scaling factor for image scaling; all line segments extracted from the image frame The values ​​are sorted in descending order.

3. The augmented reality tracking registration method according to claim 1, characterized in that: The construction of the equivalent spherical polar coordinate grid of the binocular image of the industrial scene specifically includes: constructing the equivalent spherical polar coordinate grid of the image to divide the pixels Transformed to the equivalent spherical coordinate system : , where the direction parallel to the image plane is taken as the equivalent unit sphere X Axis and Y Axis direction, Z The axis points to the optical center of the image plane, given the optical center and focal length ,by Indicates the equivalent unit sphere longitude and latitude; In the step of calculating the second vanishing point by using an orthogonal constraint, the orthogonal constraint specifically includes: , where the first vanishing point .

4. The augmented reality tracking registration method according to claim 1, characterized in that: The step of constructing a reprojection error model for the point-line feature fusion, obtaining a reprojection error based on the reprojection error model, and obtaining the optimized global pose graph based on the reprojection error specifically includes: For the reprojection error of line features, Represents the three-dimensional projection point of the detected two-dimensional line segment endpoint, so the normalized line coefficient is calculated I satisfy: (6) by Indicates in i Line segment on frame j The observation endpoint The estimated line coefficients are It represents the process of back-projecting a two-dimensional point into a three-dimensional point through the internal reference. The reprojection error of the line feature is defined as the error from the endpoint to the observation line, which is expressed as: (7) Based on the collected sensor data, the feature points of the binocular images are detected and described frame by frame and the feature points in the binocular images are matched. The attitude is estimated through the IMU data and the camera attitude is calculated using the relationship between adjacent frames. S i Indicates that it contains pose T i ,speed v i and gyroscope and accelerometer bias 、 The state variables, represents the covariance matrix of the pre-integrated rotation, velocity, and position measurement vectors, and represents the inertia residual, r ij Indicates the i-th frame and the pose x j The point feature reprojection error between point j, Represents the reprojection error of the line feature, given a set of k+1 keyframes and their states , the back-end optimization problem of fusion visual inertia is expressed as a combination of IMU residual and point line reprojection error, that is, K j To observe the key frame set of 3D points and 3D lines, the pose estimation result is optimized through the above nonlinear optimization process, and the pose and position of the binocular camera are calculated based on the feature point matching results between the binocular image of the current frame and the binocular image of the previous frame stored in the map to obtain the optimized global pose graph.

5. The augmented reality tracking registration method according to claim 1, characterized in that: Registering the target AR virtual object in the artificial marking coordinate system based on the real-time posture includes: based on the real-time posture, using Unity to draw and import the virtual object in the artificial marking coordinate system to register the target AR virtual object.

6. An augmented reality tracking and registration system, used to implement the method according to any one of claims 1 to 5, characterized in that: For industrial sites with movable binocular cameras and mobile hardware display modules, including: A coordinate transformation unit is used to move the binocular camera to collect binocular images of the industrial scene in real time, and to initialize and align the world coordinate system, the AR virtual model coordinate system, and the coordinate system of the mobile hardware platform display module based on the artificial marker, so as to construct a transformation matrix corresponding to the tracking registration result and the artificial marker coordinate system; a feature screening unit, configured to extract and match point and line features in the binocular image of the industrial scene frame by frame, screen point features in the point and line features, and screen line features based on vanishing points in the binocular image of the industrial scene, obtain a global pose graph based on the screened point and line feature set, and track and locate the target AR virtual object; Global pose graph optimization, for constructing a reprojection error model for the point-line feature fusion, obtaining a reprojection error based on the reprojection error model, and obtaining an optimized global pose graph based on the reprojection error; A target registration unit is used to estimate the real-time pose of the binocular camera in real time based on the optimized global pose graph, and register the target AR virtual object in the artificial mark coordinate system based on the real-time pose.

7. The augmented reality tracking and registration system according to claim 6, characterized in that: The binocular camera is provided with a sensor module; The coordinate transformation unit is specifically used for: Collecting real-time sensor module data of the industrial site, wherein the sensor module data includes binocular images of the industrial site scene and accelerometer and gyroscope data; Aligning the world coordinate system by artificial markers; Assume that the transformation matrix from the AR virtual model coordinate system to the artificial marker coordinate system is , the initialization alignment process is equivalent to the process of obtaining the posture of the AR virtual model relative to the display module of the mobile hardware platform, using the transformation matrix Description, meet: ； in, is the transformation matrix between the coordinate system of the artificial marker and the coordinate system of the mobile hardware platform display module, is the conversion matrix between the sensor module data and the mobile hardware platform display module, and the real-time pose estimation during the movement is expressed as , through the transformation matrix The subsequent tracking registration results can be established in the artificial identification coordinate system.

8. A computer-readable storage medium, characterized in that: The storage medium stores instructions, which, when executed, execute an augmented reality tracking and registration method according to any one of claims 1 to 5.

Images