An Augmented Reality 3D Registration Method for Complex Lighting Environments
By using adaptive thresholding and epipolar geometry feature point extraction and matching methods, combined with HoloLens2's triangulation localization, the stability and efficiency issues of 3D registration under complex lighting conditions are solved, achieving a highly efficient augmented reality experience.
Patent Information
- Application Number
- CN202310846895.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-11
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-07-11
AI Technical Summary
Existing technologies have weak resistance to changes in lighting conditions during 3D registration in complex industrial production environments, and the feature matching algorithm is inefficient, resulting in poor display effects and a poor user experience.
An image feature point extraction method with adaptive threshold, a feature point constraint matching method based on epipolar geometry, and a triangulation method suitable for HoloLens2 are adopted to achieve stable feature point extraction and efficient matching under illumination variation conditions.
It achieves stable feature point extraction and efficient matching under complex lighting conditions, resulting in a better augmented reality experience.
Smart Images

Figure CN116883630B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of 3D registration and tracking technology, and in particular to an augmented reality 3D registration method for complex lighting environments. Background Technology
[0002] With the comprehensive upgrading of global manufacturing, new technologies and manufacturing models are constantly driving the industry to move from traditional manual manufacturing towards digitalization, intelligence, and flexibility. Since the 1980s, rapidly developing augmented reality technology has been increasingly applied to various scenarios in manufacturing, such as standardized training for manual operations and real-time monitoring of production data.
[0003] Augmented reality technology mainly includes core modules such as information recognition, 3D registration and tracking, real-time rendering, human-computer interaction, and artificial intelligence, among which recognition and registration are its core research contents. The 3D registration module is responsible for matching virtual information with the real world to achieve accurate virtual-real fusion.
[0004] Combining augmented reality technology with manufacturing and applying it to actual production environments still faces challenges such as weak resistance to changes in lighting conditions in 3D registration under complex industrial production environments, high computational load under complex background conditions, and low efficiency of feature matching algorithms. These issues result in poor display effects and a poor user experience, hindering its application in various fields of industrial production. Summary of the Invention
[0005] This invention provides an augmented reality 3D registration method for complex lighting environments, based on an image feature point extraction method with adaptive thresholds, a feature point constraint matching method based on epipolar geometry, and a triangulation method suitable for HoloLen2. This method can achieve stable extraction and efficient matching of feature points under varying lighting conditions, and achieve relatively accurate position matching through triangulation, resulting in a better augmented reality experience.
[0006] This invention provides an augmented reality 3D registration method for complex lighting environments. It is based on an image feature point extraction module, a feature point constraint matching module, an object spatial localization module, HoloLens2 augmented reality glasses, and a 3D model management module. The image feature point extraction module embeds an image feature point extraction based on adaptive thresholding and non-maximum suppression to achieve stable feature point extraction under varying brightness conditions. The feature point constraint matching module embeds a feature point constraint matching algorithm based on epipolar geometry to achieve efficient matching between feature points. The object spatial localization module embeds a triangulation algorithm based on linear triangles suitable for HoloLens2 to achieve spatial localization of objects in the real scene. The HoloLens2 augmented reality glasses provide hardware and software support. The 3D model management module loads virtual object models into the matched real scene to achieve a virtual-real fusion augmented reality effect.
[0007] The method specifically includes:
[0008] S1. Obtain video stream data and store it in ObjFrame;
[0009] S2, the image feature point extraction module defines the target image, extracts and describes the features of the target region, and records the imaging pose;
[0010] S3. The feature point constraint matching module performs feature acquisition on the frame image, matches it with the target image, and determines whether the match is successful.
[0011] S4. If no match is found, return to step S1.
[0012] S5. If the match is successful, the object spatial positioning module uses the linear triangle method to solve the positioning coordinates of the feature points and displays the virtual information.
[0013] Furthermore, the specific steps for image feature point extraction by the image feature point extraction module include:
[0014] S21. Extract feature points from the image based on adaptive threshold;
[0015] S22. For the feature point extraction results obtained in step S21, perform Harris corner scoring on each candidate feature point.
[0016] S23. For the scoring results obtained in step S22, non-maximum suppression is applied to all feature points to obtain the final extraction result.
[0017] Furthermore, step S21 specifically includes:
[0018] S211. Convert the target image into a grayscale image. The image can be viewed as a two-dimensional matrix composed of the grayscale values of pixels.
[0019] S212. For each pixel in the image, calculate the gray value I of the pixels on the edge of its circular neighborhood with a radius of 3 pixels. p1 I p5 I p9 with I p13 ;
[0020] S213. Determine whether the four points obtained in step S212 simultaneously have three or more values greater than or less than the gray value of the center point. If so, the point is a candidate feature point, and its adaptive threshold is calculated using the adaptive threshold calculation formula; otherwise, the point is excluded. The specific calculation method for the adaptive threshold is as follows:
[0021]
[0022] Where I imax and I imin (i = 1, 2, ..., n) are the n values of maximum and minimum gray level in the circular neighborhood of the pixel to be tested. Experiments show that when n is 5, a good balance can be achieved between extraction effect and calculation time. σ is an adaptive parameter, and 0.3 is taken as the threshold screening standard.
[0023] S214. For feature points that satisfy the conditions of step S213, calculate the gray values of the remaining 6 pixels around their circumference.
[0024] S215. Determine whether the 6 points obtained in step S214 belong to the same class as the 3 points in step S213 using an adaptive threshold. If the condition is met, the point is a feature point; otherwise, it is excluded. The specific method for determining the class of feature points is as follows:
[0025]
[0026] Where I p I represents the grayscale value of the pixel to be measured. pn Let I be the grayscale value of the nth pixel within the circular neighborhood, and t be a fixed threshold, i.e., when I... pn ≤I p When -t, point n is considered to be darker than point p; when I pn ≥I p When +t, point n is considered to be brighter than point p; when I p -t<I pn <I p When +t, point n is considered to be a similar point to point p.
[0027] Furthermore, step S22 specifically includes:
[0028] S221. For each candidate feature point, set a rectangular sliding window on the image and move it in the x and y directions, and calculate the change in pixels within the window.
[0029] S222. Segment the image region into smooth regions, corner regions, and edge regions;
[0030] S223. Define a corner response function to clearly distinguish the pixel changes within the window across the three regions defined in step S222; specifically, the corner response function is as follows:
[0031] R H =detM-T H (tr M) 2
[0032] Where T H Let be an empirical constant, taken as (0.04, 0.06), detM be the determinant of matrix M, and trM be the trace of matrix M. The specific calculation method is as follows:
[0033]
[0034] Where λ1 and λ2 are the eigenvalues of the gradient covariance matrix M, and the gradient covariance matrix M is calculated as follows:
[0035]
[0036] Where w(x,y) is the window function of the sliding window in step S221, I x and I y These are the partial derivatives of the point (x, y) in the horizontal and vertical directions, respectively;
[0037] S224. Calculate the corner response function value for subsequent nonmaximum suppression.
[0038] Furthermore, step S23 specifically includes:
[0039] S231. Sort the global feature points according to the size of the feature point response values to obtain the feature point set P;
[0040] S232. Set the minimum suppression radius of P1 to ∞ and add it to the result set O;
[0041] S233. For each feature point P in P, P i The minimum suppression radius r is calculated. i Among them, feature point P i Minimum suppression radius r i The specific calculation method is as follows:
[0042] r i =minj |x i -x j |where R H (x i ) < cR H (x j ),x i ∈I
[0043] Where I represents the original image, R H Let c be the response function value obtained in stage S2, where c is a constant value less than 1 to ensure R H (x i ) is always less than R H (x j );
[0044] S234, P i Its minimum suppression radius r i Add the result set O, sort the result set O, discard the points outside the retention range, and continue the loop.
[0045] S235. Return the n feature points in the result set O with the smallest suppression radius in descending order as the non-maximum suppression results.
[0046] Furthermore, the specific steps of the feature point constraint matching module are as follows:
[0047] S31. Solve for the coordinates of the light intensity weight centroid in the non-uniform brightness neighborhood around each feature point, and take the direction vector from the integer point t to the gray centroid as its main direction.
[0048] S32. Scale the original image to construct an image pyramid to ensure its scale invariance;
[0049] S33. For each feature point in the image, calculate its BRIEF binary descriptor string; the specific formula for calculating the feature point BRIEF binary descriptor string is as follows:
[0050]
[0051] Where x and y are pairs of points in the neighborhood of the feature point, I x I y The grayscale values of the two points;
[0052] S34. Perform BF matching on the feature points in the original image and the target image;
[0053] S35. Filter out erroneous matching points using polar geometric constraints; the specific calculation process for the polar geometric constraints is as follows:
[0054]
[0055] The coordinates of the corresponding feature points on the two images are (x1, y1) and (x2, y2), respectively.
[0056] Furthermore, the specific process of BF matching in step S34 is as follows: traverse the n feature points of the target image, and each of them needs to be matched and calculated with all m feature points in the original image to obtain the Hamming distance between them; where the Hamming distance represents the number of different bits of characters in corresponding bits of the binary strings of two descriptors, and the smaller the value, the higher the similarity between the two feature points is considered.
[0057] Furthermore, in step S31, the formula for calculating the coordinates of the weighted centroid of light intensity is:
[0058]
[0059] The formula for calculating the direction vector angle is:
[0060]
[0061] Where N is the circular neighborhood around the feature point, m0 is the zeroth moment of the circular neighborhood, and m x and m y Let x and y be the first moment of the circular neighborhood, where x and y are the x and y coordinates of the pixels within the neighborhood. c and y c I represents the x and y coordinates of the feature points. x,y The gray value is the coordinate (x, y) point.
[0062] Furthermore, the object spatial localization module embeds a triangulation algorithm based on linear triangles suitable for Hololen2, used to achieve spatial localization of objects in real-world scenes. Specific steps include:
[0063] S51. The imaging model error is calibrated to obtain the radial distortion parameters of the HoloLens2 camera;
[0064] S52. Based on the linear triangle method, the spatial position of the object is solved by matching point pairs and the spatial coordinates of the imaging position; the specific method for solving the spatial position of the object is as follows:
[0065]
[0066] S53. Calculate image blur based on the Laplacian operator and remove blurred samples; the specific method for calculating image blur is as follows:
[0067]
[0068] in, and These are the second-order partial derivatives of the pixel at the current coordinates;
[0069] S54. Convert the positioning result from the world coordinate system to the HoloLens2 coordinate system, and render the virtual object to the correct position according to the obtained conversion result.
[0070] Furthermore, the coordinate system transformation matrix in step S54 is:
[0071]
[0072] Where (X,Y,Z) are the camera coordinates, and (s,x,y,z) are the rotation quaternions.
[0073] The beneficial effects of this invention are as follows:
[0074] 1. This invention addresses the problem that traditional feature detection methods exhibit significant differences in feature extraction results and low stability under varying light intensities in the same scene, depending on changes in ambient light. It proposes a feature extraction method based on the FAST algorithm with adaptive thresholding and non-maximum suppression, achieving stable extraction with minimal fluctuations in the number and repetition rate of extracted feature points and uniform distribution under varying light intensity. The effectiveness of the improved method has been verified.
[0075] 2. This invention, taking advantage of the HoloLens2 device's ability to acquire pose data through imaging, addresses the shortcomings of the RANSAC algorithm in the feature matching stage of the 3D registration process, such as low efficiency and high performance consumption in filtering out incorrect feature point matches. It proposes an alternative to the RANSAC method based on epipolar geometry constraints to filter out incorrect match points, thus achieving efficient constraint matching of feature points.
[0076] 3. This invention realizes a spatial object localization method using linear triangles in HoloLens2, and deploys it on HoloLens2 to perform object localization, verifying the effectiveness of the localization method and completing 3D registration. Attached Figure Description
[0077] Figure 1 This is the overall flowchart of the algorithm of this invention.
[0078] Figure 2 This is a flowchart of the feature extraction module of the present invention.
[0079] Figure 3 This is a schematic diagram of the epipolar geometry principle used in the feature matching module of this invention.
[0080] Figure 4 This is a schematic diagram of the triangulation principle of the present invention.
[0081] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0082] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0083] This invention addresses the problems of low stability in feature extraction results and low efficiency in feature matching and localization caused by changes in ambient lighting in traditional feature detection methods. By establishing adaptive thresholding and non-maximum suppression during feature point extraction, the adaptability of image feature point extraction to lighting conditions is improved. Furthermore, by establishing error filtering conditions for feature point matching based on epipolar geometry principles, the matching efficiency is improved. Finally, triangulation based on the linear triangle method is implemented on the HoloLens2 platform. The following embodiments further illustrate this invention.
[0084] This invention provides an augmented reality 3D registration method for complex lighting environments, based on an image feature point extraction module, a feature point constraint matching module, an object spatial localization module, HoloLens2 augmented reality glasses, and a 3D model management module.
[0085] The image feature point extraction module incorporates an adaptive thresholding and non-maximum suppression-based image feature point extraction method to achieve stable feature point extraction under varying brightness conditions. The feature point constraint matching module incorporates an epipolar geometry-based feature point constraint matching algorithm for efficient matching between feature points. The object spatial localization module incorporates a linear triangle-based triangulation algorithm suitable for HoloLens 2 to achieve spatial localization of objects in real-world scenes. The HoloLens 2 augmented reality glasses provide hardware and software support. The 3D model management module loads virtual object models into the matched real-world scene, achieving a seamless augmented reality effect.
[0086] like Figure 1 As shown, the method specifically includes:
[0087] S1. Obtain video stream data and store it in ObjFrame;
[0088] S2, the image feature point extraction module defines the target image, extracts and describes the features of the target region, and records the imaging pose;
[0089] S3. The feature point constraint matching module performs feature acquisition on the frame image, matches it with the target image, and determines whether the match is successful.
[0090] S4. If no match is found, return to step S1.
[0091] S5. If the match is successful, the object spatial positioning module uses the linear triangle method to solve the positioning coordinates of the feature points and displays the virtual information.
[0092] Image feature point extraction module:
[0093] In the image feature point extraction module, image feature points are extracted based on an adaptive threshold, where the n gray values I of the largest and smallest gray values in the circular neighborhood surrounding the pixel to be tested are used. imax and I imin (i = 1, 2, ..., n), the adaptive threshold is obtained by calculating the adaptive parameter σ. Achieve stable extraction of feature points under varying brightness conditions.
[0094] Meanwhile, the image feature point extraction module uses adaptive nonmaximum suppression to filter the extraction results, targeting each feature point x in the image. i Calculate its minimum suppression radius r separately. i The original image I and the corner score response function R H The minimum suppression radius r is calculated using a constant c less than 1. i =min j |x i -x j |where R H (x i ) < cR H (x j ),x i ∈I, thereby performing non-maximum suppression to achieve a uniform distribution of feature point extraction results.
[0095] like Figure 2 As shown, the specific steps for image feature point extraction by the image feature point extraction module include:
[0096] S21. Extract feature points from the image based on an adaptive threshold; Step S21 specifically includes:
[0097] S211. Acquire a template image and convert the template image (target image) into a grayscale image. The image can be viewed as a two-dimensional matrix composed of the grayscale values of pixels.
[0098] S212. For each pixel in the image, calculate the gray value I of the pixels on the edge of its circular neighborhood with a radius of 3 pixels. p1 I p5 I p9 with I p13 ;
[0099] S213. Determine whether the four points obtained in step S212 simultaneously have three or more values greater than or less than the gray value of the center point. If so, the point is a candidate feature point, and its adaptive threshold is calculated using the adaptive threshold calculation formula; otherwise, the point is excluded. The specific calculation method for the adaptive threshold is as follows:
[0100]
[0101] in and (i = 1, 2, ..., n) are the n values of maximum and minimum gray level in the circular neighborhood of the pixel to be tested. Experiments show that when n is 5, a good balance can be achieved between extraction effect and calculation time. σ is an adaptive parameter, and 0.3 is taken as the threshold screening standard.
[0102] S214. For feature points that satisfy the conditions of step S213, calculate the gray values of the remaining 6 pixels around their circumference.
[0103] S215. Determine whether the 6 points obtained in step S214 belong to the same class as the 3 points in step S213 using an adaptive threshold. If the condition is met, the point is a feature point; otherwise, it is excluded. The specific method for determining the class of feature points is as follows:
[0104]
[0105] Where I p I represents the grayscale value of the pixel to be measured. pn Let I be the grayscale value of the nth pixel within the circular neighborhood, and t be a fixed threshold, i.e., when I... pn ≤I p When -t, point n is considered to be darker than point p; when I pn ≥I p When +t, point n is considered to be brighter than point p; when I p -t<I pn <I p When +t, point n is considered to be a similar point to point p.
[0106] S22. For the feature point extraction results obtained in step S21, perform Harris corner scoring on each candidate feature point; step S22 specifically includes:
[0107] S221. For each candidate feature point, set a rectangular sliding window on the image and move it in the x and y directions, and calculate the change in pixels within the window.
[0108] S222. Segment the image region into smooth regions, corner regions, and edge regions. The specific method for segmenting the image region is as follows: For smooth regions, the ||R| obtained in step S224 is... H The value of || is small; for edge regions, R H For negative numbers with large absolute values, R is used for corner regions. H For positive numbers with relatively large values, both λ1 and λ2 have relatively large values.
[0109] S223. Define a corner response function to clearly distinguish the pixel changes within the window across the three regions defined in step S222; specifically, the corner response function is as follows:
[0110] R H =detM-T H (tr M) 2
[0111] Where T H Let be an empirical constant, taken as (0.04, 0.06), detM be the determinant of matrix M, and trM be the trace of matrix M. The specific calculation method is as follows:
[0112]
[0113] Where λ1 and λ2 are the eigenvalues of the gradient covariance matrix M, and the gradient covariance matrix M is calculated as follows:
[0114]
[0115] Where w(x,y) is the window function of the sliding window in step S221, I x and I y These are the partial derivatives of the point (x, y) in the horizontal and vertical directions, respectively;
[0116] S224. Calculate the corner response function value for subsequent nonmaximum suppression.
[0117] S23. For the scoring result obtained in step S22, non-maximum suppression is applied to all feature points to obtain the final extraction result. Step S23 specifically includes:
[0118] S231. Sort the global feature points according to the size of the feature point response values to obtain the feature point set P;
[0119] S232. Set the minimum suppression radius of P1 to ∞ and add it to the result set O;
[0120] S233. For each feature point P in P, P i The minimum suppression radius r is calculated. i Among them, feature point P i Minimum suppression radius r i The specific calculation method is as follows:
[0121] r i =min j |x i -x j |where R H (x i ) < cR H(x j ),x i ∈I
[0122] Where I represents the original image, R H Let c be the response function value obtained in stage S2, where c is a constant value less than 1 to ensure R H (x i ) is always less than R H (x j );
[0123] S234, P i Its minimum suppression radius r i Add the result set O, sort the result set O, discard the points outside the retention range, and continue the loop.
[0124] S235. Return the n feature points in the result set O with the smallest suppression radius in descending order as the non-maximum suppression results.
[0125] Feature point constraint matching module:
[0126] like Figure 3 As shown, the feature point matching process establishes epipolar geometric constraint matching conditions based on the imaging position relationship between two images. The specific operation steps are as follows: Step 1: Start the application on the Hololen2 device and extract and describe feature points frame by frame by capturing the camera video stream frames. Step 2: Calculate the image blur based on the Laplacian operator and remove blurred samples. Step 3: Perform BF matching on the feature points in the original image and the target image. Step 4: Filter out erroneous matching points through epipolar geometric constraints.
[0127] The specific steps of the feature point constraint matching module are as follows:
[0128] S31. Solve for the coordinates of the light intensity weighted centroid in the non-uniform brightness neighborhood around each feature point, and take the direction vector from the integer point t to the gray-level centroid as its principal direction; the formula for calculating the coordinates of the light intensity weighted centroid in step S31 is:
[0129]
[0130] The formula for calculating the direction vector angle is:
[0131]
[0132] Where N is the circular neighborhood around the feature point, m0 is the zeroth moment of the circular neighborhood, and m x and m y Let x and y be the first moment of the circular neighborhood, where x and y are the x and y coordinates of the pixels within the neighborhood. c and y cI represents the x and y coordinates of the feature points. x,y The gray value is the coordinate (x, y) point.
[0133] S32. Scale the original image to construct an image pyramid to ensure its scale invariance;
[0134] S33. For each feature point in the image, calculate its BRIEF binary descriptor string; the specific formula for calculating the feature point BRIEF binary descriptor string is as follows:
[0135]
[0136] Where x and y are pairs of points in the neighborhood of the feature point, I x I y The grayscale values of the two points;
[0137] S34. Perform BF matching on the feature points in the original image and the target image; the specific process of BF matching in step S34 is as follows: traverse the n feature points of the target image, and each one needs to be matched and calculated with all m feature points in the original image to obtain the Hamming distance between the two; where Hamming distance represents the number of different bits of characters in the corresponding bit positions of the binary strings of the two descriptors, and the smaller the value, the higher the similarity between the two feature points is considered.
[0138] S35. Filter out erroneous matching points using polar geometric constraints; the specific calculation process for the polar geometric constraints is as follows:
[0139]
[0140] The coordinates of the corresponding feature points on the two images are (x1, y1) and (x2, y2), respectively. Due to the inherent errors in imaging and localization, the calculation results obtained from the constraints are not zero, and the errors in the calculation results are often consistent. After the calculation is completed, the final set of matching points is obtained through cluster analysis.
[0141] Object spatial positioning module:
[0142] like Figure 4 As shown, the object spatial localization module embeds a linear triangle-based triangulation algorithm suitable for HoloLens 2, used to achieve spatial localization of objects in real-world scenes. The specific operation steps are as follows: Step 1: If the feature point constraint matching module successfully matches features, triangulation is performed using the image feature matching relationship and camera coordinates; Step 2: Based on the linear triangle method, the object's spatial position is solved by matching point pairs and the spatial coordinates of the imaging position; Step 3: The localization result is transformed from the world coordinate system to the HoloLens 2 coordinate system, and the virtual object is rendered to the correct position according to the transformed result.
[0143] The object spatial localization module embeds a triangulation algorithm based on linear triangles suitable for HoloLen2, used to achieve spatial localization of objects in real-world scenes. Specific steps include:
[0144] S51. The imaging model error is calibrated to obtain the radial distortion parameters of the HoloLens2 camera;
[0145] S52. Based on the linear triangle method, the spatial position of the object is solved by matching point pairs and the spatial coordinates of the imaging position; the specific method for solving the spatial position of the object is as follows:
[0146]
[0147] S53. Calculate image blur based on the Laplacian operator and remove blurred samples; the specific method for calculating image blur is as follows:
[0148]
[0149] in, and These are the second-order partial derivatives of the pixel at the current coordinates;
[0150] S54. Transform the positioning result from the world coordinate system to the HoloLens2 coordinate system, and render the virtual object to the correct position according to the transformed result. The coordinate transformation matrix in step S54 is:
[0151]
[0152] Where (X,Y,Z) are the camera coordinates, and (s,x,y,z) are the rotation quaternions.
[0153] This invention provides an augmented reality 3D registration method for complex lighting environments, enabling stable extraction and efficient matching of feature points under varying lighting conditions. It also achieves relatively accurate position matching through triangulation, resulting in a better augmented reality experience.
[0154] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.
[0155] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.
Claims
1. An augmented reality 3D registration method for complex lighting environments, characterized in that, Based on an image feature point extraction module, a feature point constraint matching module, an object spatial localization module, and HoloLens2 augmented reality glasses and a 3D model management module, the image feature point extraction module embeds an image feature point extraction based on adaptive thresholding and non-maximum suppression to achieve stable feature point extraction under varying brightness conditions; the feature point constraint matching module embeds a feature point constraint matching algorithm based on epipolar geometry to achieve efficient matching between feature points; the object spatial localization module embeds a triangulation algorithm based on linear triangles suitable for HoloLens2 to achieve spatial localization of objects in real-world scenes; the HoloLens2 augmented reality glasses provide hardware and software support; and the 3D model management module loads virtual object models into the matched real-world scene to achieve a virtual-real fusion augmented reality effect. The method specifically includes: S1. Obtain video stream data and store it in ObjFrame; S2, the image feature point extraction module defines the target image, extracts and describes the features of the target region, and records the imaging pose; S3. The feature point constraint matching module performs feature acquisition on the frame image, matches it with the target image, and determines whether the match is successful. S4. If no match is found, return to step S1. S5. If the match is successful, the object spatial positioning module uses the linear triangle method to solve the positioning coordinates of the feature points and displays the virtual information. The specific steps for image feature point extraction by the image feature point extraction module include: S21. Feature point extraction of the image based on adaptive threshold; specifically including: S211. Convert the target image into a grayscale image. The image is viewed as a two-dimensional matrix composed of the grayscale values of pixels. S212. For each pixel in the image, calculate the gray value of the pixels on the edge of its circular neighborhood with a radius of 3 pixels. , , and ; S213. Determine whether the four points obtained in step S212 simultaneously have three or more values greater than or less than the gray value of the center point. If so, the point is a candidate feature point, and its adaptive threshold is calculated using the adaptive threshold calculation formula; otherwise, the point is excluded. The specific calculation method for the adaptive threshold is as follows: in and These represent the maximum and minimum gray levels in the circular neighborhood of the pixel being tested, respectively. One value, We set 5, σ to be an adaptive parameter, and 0.3 to be used as the threshold screening criterion. S214. For feature points that satisfy the conditions of step S213, calculate the gray values of the remaining 6 pixels around their circumference. S215. Determine whether the 6 points obtained in step S214 belong to the same class as the 3 points in step S213 using an adaptive threshold. If the condition is met, the point is a feature point; otherwise, it is excluded. The specific method for determining the class of feature points is as follows: in The grayscale value of the pixel to be measured. The first in the circular neighborhood grayscale value of each pixel. For a fixed threshold, that is, when At that time, it was believed The point is more Dark spot; when At that time, it was believed The point is more Bright spot; when At that time, it was believed The point is Similarities; S22. For the feature point extraction results obtained in step S21, perform Harris corner scoring on each candidate feature point. S23. For the scoring results obtained in step S22, non-maximum suppression is applied to all feature points to obtain the final extraction results; specifically including: S231. Sort the global feature points according to the magnitude of their response values to obtain the feature point set. ; S232, will Minimum suppression radius set to Add to result set ; S233, Regarding Each feature point in Calculate its minimum suppression radius. Among them, feature points Minimum suppression radius The specific calculation method is as follows: in For the original image, The response function value obtained in stage S2. It is a constant value less than 1, used to ensure Always less than ; S234, will Its minimum suppression radius Add to result set and the result set Sort the data, discard points outside the retention range, and continue the loop. S235, Return the result set Minimum suppression radius from largest to smallest The feature points are used as the result of nonmaximum suppression; The specific steps of the feature point constraint matching module are as follows: S31. Solve for the coordinates of the light intensity weight centroid in the non-uniform brightness neighborhood around each feature point, and take the direction vector from the feature point to the gray centroid as its main direction. S32. Scale the original image to construct an image pyramid to ensure its scale invariance; S33. For each feature point in the image, calculate its BRIEF binary descriptor string; the specific formula for calculating the feature point BRIEF binary descriptor string is as follows: in , For a pair of points within the neighborhood of a feature point, , The grayscale values of the two points; S34. Perform BF matching on the feature points in the original image and the target image; S35. Filter out erroneous matching points using polar geometric constraints; the specific calculation process for the polar geometric constraints is as follows: The coordinates of the corresponding feature points in the two images are respectively and .
2. The augmented reality 3D registration method for complex lighting environments according to claim 1, characterized in that, Step S22 specifically includes: S221. For each candidate feature point, set a rectangular sliding window on the image, and... and Move in the direction and calculate the change in pixels within the window; S222. Segment the image region into smooth regions, corner regions, and edge regions; S223. Define a corner response function to clearly distinguish the pixel changes within the window across the three regions defined in step S222; specifically, the corner response function is as follows: in It is an empirical constant. For matrix The determinant, For matrix The trace is calculated as follows: in and The gradient covariance matrix eigenvalues, gradient covariance matrix The specific calculation method is as follows: in For the window function of the sliding window in step S221, and Points Partial derivatives in both the horizontal and vertical directions; S224. Calculate the corner response function value for subsequent nonmaximum suppression.
3. The augmented reality 3D registration method for complex lighting environments according to claim 1, characterized in that, The specific process of BF matching in step S34 is as follows: traversing the target image. Each of the 10 feature points must be correlated with all the feature points in the original image. The Hamming distance between two feature points is calculated by matching each feature point. The Hamming distance represents the number of different bits in the corresponding bits of the binary strings of the two descriptors. The smaller the value, the higher the similarity between the two feature points.
4. The augmented reality 3D registration method for complex lighting environments according to claim 1, characterized in that, In step S31, the formula for calculating the centroid coordinates of the light intensity weight is: The formula for calculating the direction vector angle is: in, The circular neighborhood around the feature point For the zeroth moment of the circular neighborhood, and Let the first moment be the circular neighborhood. and The x and y coordinates of the pixels in the neighborhood. and The x and y coordinates of the feature points are: coordinates The grayscale value of a point.
5. The augmented reality 3D registration method for complex lighting environments according to claim 1, characterized in that, The object spatial localization module embeds a triangulation algorithm based on linear triangles suitable for HoloLen2, used to achieve spatial localization of objects in real-world scenes. Specific steps include: S51. The imaging model error is calibrated to obtain the radial distortion parameters of the HoloLens2 camera; S52. Based on the linear triangle method, the spatial position of the object is solved by matching point pairs and the spatial coordinates of the imaging position; the specific method for solving the spatial position of the object is as follows: S53. Calculate image blur based on the Laplacian operator and remove blurred samples; the specific method for calculating image blur is as follows: in, , and These are the second-order partial derivatives of the pixel at the current coordinates; S54. Convert the positioning result from the world coordinate system to the HoloLens2 coordinate system, and render the virtual object to the correct position according to the obtained conversion result.
6. The augmented reality 3D registration method for complex lighting environments according to claim 5, characterized in that, The coordinate system transformation matrix in step S54 is: in, For camera coordinates, It is a rotation quaternion.
