An ar tracking registration method of optical flow tracking and mean shift and related device

Abstract

The application discloses an AR tracking registration method based on optical flow tracking and mean shift and a related device, relates to the technical field of AR three-dimensional tracking registration, and comprises the following steps: acquiring a video stream, selecting an initial frame, and framing and selecting a target region to be registered as a template image; performing AR feature detection matching on the template image and a current frame of the video stream to obtain a feature point matching pair; purifying and initializing the matching pair to obtain an initial feature point set of the current frame optical flow and a three-dimensional coordinate point set thereof; tracking the target region frame by frame by using a pyramid optical flow to update two-dimensional coordinates of the feature points; when the optical flow tracking is lost, the target region is relocated and the tracking is resumed through mean shift; camera poses are solved according to the updated two-dimensional feature coordinates and the three-dimensional coordinate point set, and AR virtual content is rendered in real time based on the poses, so that the speed and stability of AR three-dimensional tracking registration can be improved, and fast relocation after target loss is realized.

Description

Technical Field

[0001] This application relates to the field of AR 3D tracking and registration technology, and in particular to an AR tracking and registration method and related apparatus using optical flow tracking and mean shift. Background Technology

[0002] AR overlays virtual information onto the real world, enhancing its appearance. Mainstream visual feature-based augmented reality 3D tracking and registration methods achieve virtual-real fusion registration through feature point extraction, matching, and tracking for pose estimation. However, existing methods suffer from low computation speed, poor stability, and susceptibility to complex environments.

[0003] Therefore, improving the speed and stability of AR 3D tracking registration and achieving rapid relocation after target loss are urgent technical problems that need to be solved. Summary of the Invention

[0004] The purpose of this application is to provide an AR tracking registration method and related apparatus based on optical flow tracking and mean shift, which can improve the speed and stability of AR 3D tracking registration and achieve rapid relocation after target loss.

[0005] To achieve the above objectives, this application provides the following solution: Firstly, this application provides an AR tracking registration method based on optical flow tracking and mean shift, including: Acquire the video stream, select the initial frame in the video stream, and select the target area to be registered as the template image.

[0006] AR feature detection and matching are performed on the template image and the current frame image in the video stream to obtain several feature point matching pairs.

[0007] The feature point matching pairs are purified and initialized to obtain the initial set of optical flow feature points and the corresponding three-dimensional spatial coordinate point set of the current frame.

[0008] The pyramid optical flow frame-by-frame tracking method is used to track the target region of the current frame and update the two-dimensional coordinates of the initial optical flow feature point set in the current frame. When the target region of the current frame tracked by the pyramid optical flow frame-by-frame tracking method is lost, the target region is relocated by mean shift and the optical flow tracking is restored.

[0009] The camera pose of the current frame is solved based on the updated initial optical flow feature point set in the two-dimensional coordinates of the current frame and the three-dimensional spatial coordinate point set.

[0010] Real-time rendering of AR virtual content is performed in the corresponding area of ​​the current frame based on the camera pose.

[0011] Secondly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the AR tracking and registration method for optical flow tracking and mean shift described above.

[0012] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the AR tracking and registration method for optical flow tracking and mean shift described above.

[0013] Fourthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the AR tracking and registration method for optical flow tracking and mean shift described above.

[0014] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides an AR tracking registration method and related apparatus based on optical flow tracking and mean shift. By acquiring a video stream, an initial frame is selected, and the target region to be registered is bounded as a template image. AR feature detection and matching are then performed on the template image and the current frame image, solving the problem of rapid target region definition and cross-frame feature association. This achieves zero-sample template initialization without offline training, providing a reliable sparse feature prior for subsequent tracking. By refining and initializing the feature point matching pairs, an initial set of optical flow feature points and corresponding 3D spatial coordinate points are obtained. The pyramid optical flow frame-by-frame tracking method is used to update the 2D coordinates of the feature points in the current frame. Then, based on the updated 2D coordinates and 3D spatial coordinate points, the camera pose in the current frame is solved. This solves the coupling problem between 2D tracking and 3D pose estimation in dynamic scenes, achieving smooth inter-frame tracking of the optical flow feature point set and real-time joint calculation of the camera's six-degree-of-freedom pose. While ensuring lightweight computation, it effectively handles large displacements and scale changes, providing a frame-level stable geometric alignment benchmark for virtual-real fusion. When the target region tracked frame by frame by pyramid optical flow is lost, the target region is relocated and optical flow tracking is resumed by mean drift. This solves the problem of irreversible loss of optical flow caused by occlusion or rapid movement, and realizes fast recapture based on color distribution without feature dependence. Together with the main optical flow tracking, it forms a closed-loop mechanism of normal tracking plus abnormal self-healing, which significantly improves the system's survivability in long-term complex scenarios and avoids the whole system restart due to a single tracking failure. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is an application environment diagram of an AR tracking registration method with optical flow tracking and mean shift according to an embodiment of this application; Figure 2 A flowchart illustrating an AR tracking registration method with optical flow tracking and mean shift provided in an embodiment of this application; Figure 3 A schematic diagram of the overall process of an AR tracking registration method with optical flow tracking and mean shift provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] The optical flow tracking and mean shift AR tracking registration method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be set up independently, integrated into server 104, or placed in the cloud or on another server. Terminal 102 can send the video stream to be processed to server 104. After receiving the video stream, server 104 acquires the video stream, selects the initial frame in the video stream, and frames the target area to be registered as a template image. AR feature detection and matching are performed on the template image and the current frame image in the video stream to obtain several feature point matching pairs. These feature point matching pairs are then refined and initialized to obtain the initial optical flow feature point set and the corresponding three-dimensional spatial coordinate point set for the current frame. A pyramid optical flow frame-by-frame tracking method is used to track the target area of ​​the current frame, updating the two-dimensional coordinates of the initial optical flow feature point set in the current frame. When the target area of ​​the current frame tracked by the pyramid optical flow frame-by-frame tracking method is lost, mean shift repositioning is performed to relocate the target area and optical flow tracking is resumed. Based on the updated two-dimensional coordinates of the initial optical flow feature point set in the current frame and the three-dimensional spatial coordinate point set, the camera pose of the current frame is solved. Real-time rendering of AR virtual content is performed according to the corresponding area of ​​the current frame based on the camera pose. Server 104 can then feed back the obtained rendering results to terminal 102. In addition, in some embodiments, the AR tracking registration method of optical flow tracking and mean shift can also be implemented by the server 104 or the terminal 102 separately. For example, the terminal 102 can directly perform AR tracking registration on the video stream to be processed, or the server 104 can obtain the video stream to be processed from the data storage system and perform AR tracking registration on the video stream to be processed.

[0020] The terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster composed of multiple servers, or it can be a cloud server.

[0021] In one exemplary embodiment, such as Figure 2 and Figure 3 As shown, an AR tracking registration method based on optical flow tracking and mean shift is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps 201 to 206: Step 201: Obtain the video stream, select the initial frame in the video stream, and select the target area to be registered as the template image.

[0022] Step 202: Perform AR feature detection and matching on the template image and the current frame image in the video stream to obtain several feature point matching pairs.

[0023] Step 203: Purify and initialize several feature point matching pairs to obtain the initial set of optical flow feature points and the corresponding three-dimensional spatial coordinate point set of the current frame.

[0024] Step 204: Use the pyramid optical flow frame-by-frame tracking method to track the target region of the current frame and update the two-dimensional coordinates of the initial optical flow feature point set in the current frame; when the target region of the current frame tracked by the pyramid optical flow frame-by-frame tracking method is lost, perform mean drift relocation to relocate the target region and resume optical flow tracking.

[0025] Step 205: Based on the updated initial optical flow feature point set in the current frame's two-dimensional coordinates and the three-dimensional spatial coordinate point set, solve for the camera pose of the current frame.

[0026] Step 206: Real-time rendering of AR virtual content in the corresponding area of ​​the current frame based on the camera pose.

[0027] By implementing steps 201 to 206 above, this application combines initial pose estimation based on feature matching with frame-by-frame tracking based on pyramid optical flow, and introduces a mean drift relocation mechanism when the target area is lost. This solves the problems of large computational load and significant inter-frame jitter in the pure feature matching method, and also makes up for the defects of easy drift and inability to recover tracking in the pure optical flow method. Thus, while ensuring real-time performance, it significantly improves the tracking stability and anti-interference ability of the target area in complex scenes. At the same time, through the continuous correlation and update of the optical flow feature point set and the three-dimensional spatial coordinate point set, the continuity and accuracy of the camera pose solution for each frame are ensured. Finally, it realizes accurate, smooth and persistent augmented reality overlay display of virtual content in real scene, achieving an overall technical effect of high efficiency, robustness and self-recovery.

[0028] In another exemplary embodiment of this application, in order to accurately perform AR feature detection and matching on the template image and the current frame image to obtain several feature point matching pairs, the above step 202 is replaced by the following steps 2021 to 2023: Step 2021: Use the ORB algorithm to perform feature point detection on the template image and the current frame image to obtain the feature point set of the template image and the feature point set of the current frame image.

[0029] The specific operation is as follows: For a certain pixel in the image , with radius Construct a circle, if there exists on the circle consecutive pixels satisfy The following grayscale difference conditions are used to determine the pixel point. For feature points, where Represents pixels pixel grayscale values, The grayscale difference threshold; Represents pixels The pixel grayscale values ​​are then used. Based on this, an 8-layer image pyramid is constructed with a scaling factor of 1.2, and FAST feature point detection is performed on each layer to impart scale invariance to the feature points, ensuring stable detection even when the target scale changes. Subsequently, the Harris response function is used to sort all detected feature points, retaining the top ones with the largest response values. Weak corner points are removed by using a set of 'points', and the positions of the feature points in scale space are accurately fitted using two-dimensional quadratic function interpolation and one-dimensional function interpolation to obtain the feature point set of the template image and the feature point set of the current frame image. The feature point set output in this step is directly used as the input for subsequent FREAK descriptor calculation and optical flow tracking initialization.

[0030] Step 2022: Use FREAK binary descriptors to describe the feature points of the template image and the feature points of the current frame image respectively, to obtain the template image descriptor and the current frame image descriptor.

[0031] The specific operation is as follows: For each feature point detected in step 2021, a FREAK binary descriptor is used for feature description. A circular sampling structure with different Gaussian kernel standard deviations is established centered on the feature point. Gaussian blur is applied to each sampling point; the farther the sampling point is from the feature point, the larger the Gaussian blur radius. Binary codes are generated by comparing the gray values ​​of sampling point pairs. The expression for the binary code is: ; ; in, Represents any pair of sampling points; The number of sample point pairs constructed is the length of the binary descriptor; and This represents the grayscale value after Gaussian smoothing of the sampled points. If the difference in grayscale values ​​is greater than 0, the corresponding position of the descriptor will be set to 1; otherwise, the position will be set to 0. and These are two specific sampling points (pixel positions) used for grayscale comparison or gradient calculation. In addition, 45 symmetrical sampling point pairs were selected and their local gradient averages were calculated. The gradient average is calculated by assigning directional information to the descriptor as follows: ; in, This represents the total number of sampling point pairs; This represents the set of selected sampling point pairs, with each pair containing two sampling points; and These are the coordinates of the two sampling points.

[0032] Step 2023: For each feature point in the template image feature point set, search for candidate matching points in the current frame image. Use a coarse-to-fine retinal scan strategy to compare the template image descriptor with the current frame image descriptor to obtain feature point matching pairs. The coarse-to-fine retinal scan strategy is as follows: first compare the first few bits of the template image descriptor with the current frame image descriptor. If the Hamming distance of the first few bits is less than a second preset threshold, continue to compare the remaining bits to obtain the complete Hamming distance. If the Hamming distance of the first few bits is greater than or equal to the second preset threshold, directly discard the current candidate matching point pair.

[0033] Specifically, a coarse-to-fine retinal scan matching search is employed, first selecting sample pairs in the outer ring, then selecting sample pairs in the inner ring. The first 128 bits of the two descriptors to be matched are compared; if they are less than a set threshold... If the first 128 bits of the two descriptors are not matched, then the matching process continues. Since more than 90% of the candidates are discarded when comparing the first 128 bits of the descriptors, the matching efficiency is significantly improved.

[0034] In another exemplary embodiment of this application, in order to accurately purify and initialize the plurality of feature point matching pairs to obtain the initial set of optical flow feature points and the corresponding three-dimensional spatial coordinate point set of the current frame, the above step 203 is replaced by the following steps 2031 to 2034: Step 2031: Remove feature point matching pairs whose Hamming distance is greater than the third preset threshold to obtain the first set of filtered matching pairs.

[0035] Specifically, feature point pairs with a Hamming distance greater than the threshold of 100 in step 202 are identified as incorrect matches and removed from subsequent RANSAC iterations. Experiments show that a large number of incorrect matches occur when the Hamming distance is greater than 100, and setting the threshold to 100 yields the best results. This significantly reduces the proportion of incorrect points entering RANSAC, thereby reducing the number of iterations required and improving overall efficiency.

[0036] Step 2032: Sort the feature point matching pairs in the first filtered matching pair set in ascending order of Hamming distance, and use the sequential sampling RANSAC algorithm to purify the sorted feature point matching pairs to obtain the purified feature point matching pair set and homography matrix.

[0037] Specifically, the feature point matching pairs in the first set of filtered matching pairs are sorted in ascending order of Hamming distance, resulting in an ordered sequence of similarity from high to low. During the RANSAC iteration process, the... m The next iteration selects the {4th}th element from the ordered sequence in sequence. m -1)+1,4( m -1)+2,4( m -1)+3,4( m -1)+4} four pairs of non-collinear feature points are used to estimate the initial homography matrix. H ,use H Test the remaining matching point pairs and calculate the reprojection error; if the error is less than the threshold... d The point pairs are added to the current inlier set, and the upper limit of the iteration count is dynamically adjusted based on the confidence level and the current number of inliers. This process is repeated. M After the second iteration, select the interior point with the largest number of values. H As the optimal result, ascending order ensures that the point pairs with the highest similarity are selected first in each iteration, while sequential sampling ensures that different point pairs are used in each iteration. The combination of these two methods avoids premature convergence that might be caused by random sampling, thus improving the accuracy and stability of homography matrix estimation. The optimal output of this step is... H The purified feature point matching pair set provides attitude estimation basis to the mean drift module and initial feature point sequence to the optical flow tracking module.

[0038] Step 2033: Extract feature points located on the current frame image from the purified feature point matching pair set, calculate the pixel distance between each feature point, remove feature points with a pixel spacing less than the fourth preset threshold, obtain the initial optical flow feature point set, and record the two-dimensional coordinates of each feature point.

[0039] Specifically, before optical flow tracking begins, feature points located in the current frame image from each feature point matching pair are extracted. Distance constraints are applied to the extracted feature points: ensuring that the distance between any two feature points is greater than a set threshold; if the distance between two feature points is less than the threshold, spatial clustering is considered to exist, and these feature points are removed from the sequence. This results in a uniformly distributed initial set of optical flow feature points, with the two-dimensional coordinates of each feature point recorded as Points_0. This uniform distribution improves the stability of optical flow tracking and reduces redundant computation in subsequent PnP solutions.

[0040] Step 2034: Based on the two-dimensional coordinates of each feature point in the current frame image in the initial optical flow feature point set, and combined with the homography matrix and camera intrinsic parameters, calculate the three-dimensional spatial coordinates corresponding to each feature point to obtain the corresponding three-dimensional spatial coordinate point set.

[0041] Specifically, based on the coordinates of each two-dimensional feature point in Points_0, and combined with the known spatial geometric relationships of the target, the corresponding three-dimensional spatial coordinate point Points3D for each feature point is calculated and recorded, establishing a 2D-3D point correspondence. Simultaneously, the feature points and descriptors of the template image are saved for pose recovery in case of feature point loss.

[0042] As an optional implementation, a pyramid optical flow method (a multi-scale version of the LK optical flow method) is used to track Points_0 frame by frame. By using a multi-scale image pyramid to handle large displacement motions, the robustness of feature point tracking is improved, while avoiding global feature detection for each frame, significantly reducing the computational load. During the tracking process, the sequence of lost feature points (those that are out of bounds or occluded and cannot be tracked) is recorded simultaneously to obtain the new position coordinates of each feature point in the current frame (the updated Points_0).

[0043] The updated Points_0 and the corresponding Points3D are directly input into the solvePnP function to solve for the camera extrinsic parameters (rotation matrix R and translation vector t) of the current frame. This process does not rely on the accumulation of pose changes between adjacent frames, but instead solves for the absolute pose directly from the 2D-3D correspondence for each frame, completely eliminating the accumulation of inter-frame errors and ensuring the stability of long-term tracking. The solved camera extrinsic parameters of the current frame are combined with the camera intrinsic parameters for the final virtual model rendering and overlay output.

[0044] As an optional implementation, when the target region tracked by the pyramid optical flow frame-by-frame tracking method in the current frame is lost, mean-shift relocation of the target region and restoration of optical flow tracking are performed, specifically including the following steps: Step 301: When the target region of the current frame tracked by the pyramid optical flow frame-by-frame tracking method is lost, the background weighted mean drift algorithm is called to relocate the target region in the local area of ​​the current frame based on the target color histogram pre-generated from the template image.

[0045] Step 302: Re-execute feature detection and matching within the relocated target area to obtain the initial feature point set and the corresponding three-dimensional spatial coordinate point set, and return to step "Use the pyramid optical flow frame-by-frame tracking method to track the target area of ​​the current frame and update the two-dimensional coordinates of the initial optical flow feature point set in the current frame".

[0046] A background-weighted mean shift (CBWH) algorithm is employed to track the target region frame by frame. CBWH introduces background color histogram weighting on top of traditional mean shift, assigning a background weighting coefficient to each pixel to reduce the interference of background color on the target histogram and improve region tracking accuracy against complex backgrounds. Using the target color histogram as the target model and the background region color histogram as the interference reference, iterative mean shift converges to the center of the target region, outputting the position and size of the target in the current frame (tracking window). The tracking window directly serves as the limiting input for feature point detection, constraining feature point detection within the target region, reducing background interference and computational cost.

[0047] Within the target region (tracking window) obtained by CBWH tracking, an improved feature point detection and matching algorithm is invoked. Feature point detection and matching are performed only within the target region. The optimal homography matrix between the target region and the template image in the current frame is estimated, and pyramid optical flow is returned for frame-by-frame tracking. By limiting the detection range to the target region, interference from feature points in the background region is avoided, reducing the false matching rate.

[0048] It should be noted that after each frame of pyramid optical flow tracking is completed, it is determined whether the number of currently effective tracked feature points is less than a threshold of 8. If it is less than the threshold of 8, the target region is determined to be lost. Within the relocated target region, Points_0 and the corresponding Points3D are reinitialized to restore the optical flow tracking and attitude solution of subsequent frames with an accurate initial state, forming a closed-loop mechanism of "tracking → loss detection → recovery → continued tracking".

[0049] As an optional implementation, if mean-shift relocation fails, global template matching relocation is performed based on the template image.

[0050] If global template matching relocation is successful, feature detection and matching are re-executed within the relocated target area to obtain the initial feature point set and the corresponding three-dimensional spatial coordinate point set, and the process returns to the step "Use the pyramid optical flow frame-by-frame tracking method to track the target area of ​​the current frame and update the two-dimensional coordinates of the initial optical flow feature point set in the current frame".

[0051] If global template matching relocation fails, optical flow tracking is not performed on the current frame. The system waits for the next frame and uses the next frame as the current frame, returning to the step "perform AR feature detection and matching on the template image and the current frame image in the video stream to obtain several feature point matching pairs".

[0052] As an optional implementation, the global template matching relocation specifically includes the following: Using the template image as a matching template, the similarity between the template image and the corresponding image region at each matching position in the current frame is calculated within the global range of the current frame or within a preset search area.

[0053] If there is a location with a similarity greater than the first preset threshold, the location with the highest similarity is selected as the relocation result of the target region.

[0054] Specifically, when the mean drift tracking result is abnormal (such as target loss or tracking window drifting to the wrong area), template matching relocation is initiated: using the template image as a reference, the template matching method is used to search for the best matching position of the target in the global range of the current frame (or the expanded search area), the target area is relocated, and the initial window position of mean drift tracking is restored.

[0055] As an optional implementation, the process of mean-shift relocation of the target region also includes updating the target color histogram model: Obtain the color histogram of the target region tracked in the current frame.

[0056] The color histogram of the target region in the current frame is weighted and fused with the target color histogram of the template image using a preset weight. The fusion result is used as the updated target color histogram model for mean drift tracking in subsequent frames.

[0057] Specifically, during continuous tracking, based on the target region tracked in the current frame, the target color histogram model is adaptively updated using a weighted fusion of the current frame's target region color histogram and the original template histogram. This allows the target model to adapt to gradual changes in the target's appearance over time (such as changes in illumination or viewing angle), preventing tracking failure due to appearance drift. The weighted fusion method retains the original template features while incorporating information from the current frame, avoiding target drift caused by excessive model updates. The updated target model is directly used as the input to the target histogram for the next frame, forming a continuous adaptive update closed loop for the model.

[0058] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 4As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data used in the AR tracking registration method that implements optical flow tracking and mean shift. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements an AR tracking registration method that implements optical flow tracking and mean shift.

[0059] Figure 4 The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0060] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0061] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0062] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations and be authorized by the owner of the corresponding device.

[0063] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0064] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0065] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0066] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An AR tracking registration method using optical flow tracking and mean shift, characterized in that, The method includes: Acquire the video stream, select the initial frame in the video stream, and use the target area to be registered as the template image; AR feature detection and matching are performed on the template image and the current frame image in the video stream to obtain several feature point matching pairs; The feature point matching pairs are purified and initialized to obtain the initial set of optical flow feature points and the corresponding three-dimensional spatial coordinate point set of the current frame. The pyramid optical flow frame-by-frame tracking method is used to track the target region of the current frame and update the two-dimensional coordinates of the initial optical flow feature point set in the current frame. When the target region of the current frame tracked by the pyramid optical flow frame-by-frame tracking method is lost, mean drift is performed to relocate the target region and restore optical flow tracking. Based on the updated initial optical flow feature point set in the current frame's two-dimensional coordinates and the three-dimensional spatial coordinate point set, the camera pose of the current frame is solved; Real-time rendering of AR virtual content is performed in the corresponding area of ​​the current frame based on the camera pose.

2. The AR tracking and registration method based on optical flow tracking and mean shift according to claim 1, characterized in that, When the target region of the current frame tracked using the pyramid optical flow frame-by-frame tracking method is lost, mean-shift relocation of the target region is performed and optical flow tracking is resumed, specifically including: When the target region of the current frame tracked by the pyramid optical flow frame-by-frame tracking method is lost, the background weighted mean drift algorithm is called to relocate the target region in the local area of ​​the current frame based on the target color histogram pre-generated from the template image. In the relocated target area, feature detection and matching are re-executed to obtain the initial feature point set and the corresponding three-dimensional spatial coordinate point set. Then, the process returns to the step "Use the pyramid optical flow frame-by-frame tracking method to track the target area of ​​the current frame and update the two-dimensional coordinates of the initial optical flow feature point set in the current frame".

3. The AR tracking and registration method based on optical flow tracking and mean shift according to claim 2, characterized in that, If mean drift relocation fails, then global template matching relocation is performed based on the template image; If global template matching relocation is successful, feature detection and matching will be re-executed within the relocated target area to obtain the initial feature point set and the corresponding three-dimensional spatial coordinate point set, and the process will return to the step "Use the pyramid optical flow frame-by-frame tracking method to track the target area of ​​the current frame and update the two-dimensional coordinates of the initial optical flow feature point set in the current frame". If global template matching relocation fails, optical flow tracking is not performed on the current frame. The system waits for the next frame and uses the next frame as the current frame, returning to the step "perform AR feature detection and matching on the template image and the current frame image in the video stream to obtain several feature point matching pairs".

4. The AR tracking and registration method based on optical flow tracking and mean shift according to claim 3, characterized in that, The global template matching relocation specifically includes: Using the template image as a matching template, the similarity between the template image and the corresponding image region at each matching position in the current frame is calculated within the global range of the current frame or within a preset search area. If there is a location with a similarity greater than the first preset threshold, the location with the highest similarity is selected as the relocation result of the target region.

5. The AR tracking and registration method based on optical flow tracking and mean shift according to claim 2, characterized in that, The process of mean-shift relocation of the target region also includes updating the target color histogram model: Obtain the color histogram of the target region tracked in the current frame; The color histogram of the target region in the current frame is weighted and fused with the target color histogram of the template image using a preset weight. The fusion result is used as the updated target color histogram model for mean drift tracking in subsequent frames.

6. The AR tracking and registration method based on optical flow tracking and mean shift according to claim 1, characterized in that, AR feature detection and matching are performed on the template image and the current frame image to obtain several feature point matching pairs, specifically including: The ORB algorithm is used to detect feature points in the template image and the current frame image to obtain the feature point set of the template image and the feature point set of the current frame image. The FREAK binary descriptor is used to describe the feature points of the template image and the feature points of the current frame image respectively, so as to obtain the template image descriptor and the current frame image descriptor; For each feature point in the template image feature point set, candidate matching points are searched in the current frame image. A coarse-to-fine retinal scan strategy is adopted to compare the template image descriptor with the current frame image descriptor to obtain feature point matching pairs. The coarse-to-fine retinal scan strategy is as follows: first, the first few bits of the template image descriptor and the current frame image descriptor are compared. If the Hamming distance of the first few bits is less than a second preset threshold, the remaining bits are compared to obtain the complete Hamming distance. If the Hamming distance of the first few bits is greater than or equal to the second preset threshold, the current candidate matching point pair is directly discarded.

7. The AR tracking and registration method based on optical flow tracking and mean shift according to claim 1, characterized in that, The feature point matching pairs are purified and initialized to obtain the initial set of optical flow feature points and the corresponding three-dimensional spatial coordinate point set of the current frame, specifically including: Remove feature point matching pairs whose Hamming distance is greater than the third preset threshold to obtain the first set of filtered matching pairs; The feature point matching pairs in the first filtered matching pair set are sorted in ascending order of Hamming distance. The sequential sampling RANSAC algorithm is used to purify the sorted feature point matching pairs to obtain the purified feature point matching pair set and homography matrix. Extract the feature points located on the current frame image from the purified feature point matching pair set, calculate the pixel distance between each feature point, remove feature points with a pixel distance less than the fourth preset threshold, obtain the initial feature point set of optical flow, and record the two-dimensional coordinates of each feature point; Based on the two-dimensional coordinates of each feature point in the current frame image in the initial optical flow feature point set, and combined with the homography matrix and camera intrinsic parameters, the three-dimensional spatial coordinates corresponding to each feature point are calculated to obtain the corresponding three-dimensional spatial coordinate point set.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that the processor executes the computer program to implement the AR tracking registration method of optical flow tracking and mean shift as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the AR tracking registration method of optical flow tracking and mean shift as described in any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the AR tracking registration method of optical flow tracking and mean shift as described in any one of claims 1-7.

Images