Text scene-oriented augmented reality tracking registration method, system and storage medium

Abstract

A kind of text scene-oriented augmented reality tracking registration method, system and storage medium, method includes input camera shot video frame, current frame is divided into ordinary frame, tracking frame, key frame by frame decision rule;Key frame is sequentially carried out image feature extraction, feature matching, pose estimation obtains camera pose and is tracked and registered, image feature extraction is detected according to rule respectively the stroke end point, turning point and trifurcation point of character;Using optical flow method, the pose corresponding to the tracking frame is combined with the pose corresponding to the last tracking frame or key frame, the pose of current tracking frame is calculated, and registration is carried out;The average change of each frame is obtained by solving the change quantity of the two poses and the difference of the intermediate frame number between the tracking frame and the key frame with known pose, so that the pose of the ordinary frame linearly changes from the previous pose to the next pose.The application can improve the text image feature extraction capability, so that the feature matching result has higher accuracy, and the tracking registration processing time is shortened.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of augmented reality, and particularly relates to an augmented reality tracking registration method for a text scene, a system and a storage medium. BACKGROUND

[0002] Augmented reality (AR) technology is a technology of capturing real world image information by using a camera device and superimposing and fusing generated virtual information with a real scene, which can improve the clarity, intuitiveness and perceptual impact of the real scene and has been increasingly applied in the education field to assist users in learning. Tracking registration technology is a key technology for determining the performance of an AR system, and the purpose thereof is to quickly and accurately calculate the pose information of a camera relative to a real scene and accurately align virtual information with the real scene according to the pose information. Most of the existing tracking registration technologies adopt a natural feature-based manner to identify a target, and complete virtual-real fusion through feature extraction, matching and tracking for pose estimation. However, when AR is applied to an offline learning scene, a text on a textbook often needs to be tracked and registered as an identification target. Due to the limitation of device hardware resources and the weak texture features of a text image, the existing method has the problems of poor feature extraction effect, low tracking registration accuracy and poor real-time performance. Therefore, a method capable of quickly tracking a text on a textbook and accurately registering virtual information to a corresponding target plays a key role in applying AR to an offline textbook learning scene. SUMMARY

[0003] The application aims to solve the problems in the prior art, and provides an augmented reality tracking registration method for a text scene, a system and a storage medium, which improves the text image feature extraction capability, has a higher accuracy rate in feature matching results, shortens the tracking registration processing time and improves the real-time performance.

[0004] In order to achieve the above-mentioned purpose, the application has the following technical solutions:

[0005] An augmented reality tracking registration method for a text scene comprises the following steps:

[0006] Input a video frame captured by a camera, and divide the current frame into an ordinary frame, a tracking frame and a key frame through a frame decision rule;

[0007] Track and register the key frame by sequentially performing image feature extraction, feature matching and pose estimation to obtain the camera pose, wherein the image feature extraction detects the stroke endpoints, turning points and three-way points of the text according to the following mathematical expressions:

[0008] The stroke endpoints satisfy:

[0009]

[0010] Stroke turning point satisfies:

[0011]

[0012] Stroke three-way point satisfies:

[0013]

[0014] Wherein, is a set of sampling points with similar gray values, is a set of sampling points with large gray value differences, represents the number of elements in the set represents the number of elements in the set , , , , are respectively a set of adjacent sampling points with similar gray values, are respectively a set of adjacent sampling points with large gray value differences, represents that the front and rear expressions are satisfied simultaneously;

[0015] Using the optical flow method, the pose of the current tracking frame is calculated by combining the pose corresponding to the last tracking frame or key frame, and registration is performed;

[0016] By calculating the average change amount of each frame from the pose change amount of the tracking frame and the key frame and the difference in the number of intermediate frames, the pose of the ordinary frame is linearly changed from the previous pose to the next pose, and the augmented reality tracking registration of all video frames is completed.

[0017] As a preferred solution, in the step of dividing the current frame into ordinary frames, tracking frames and key frames by the frame decision rule:

[0018] The first frame or key frame counter Counter key When the input frame is equal to 0, it is a key frame; by initializing Counter key the value, that is, a key frame is taken every set number of frames, and different values can be given according to the device performance; every frame is processed, including tracking frames and ordinary frames, Counter key decreased by one, in the case of special circumstances Counter key will be directly modified, that is, not fixed to take a key frame every set number of frames; after key frame processing, Counter key reset to the initialization value;

[0019] the counterCounter trk Input frame as tracking frame when equal to 0; Counter trk Initial value is 1, and the value is reduced by 1 after processing one normal frame; when a key frame or tracking frame occurs, the value is reset to 1. Counter trk Reset to 1;

[0020] Except for key frames and tracking frames, the remaining frames are normal frames.

[0021] As a preferred solution, the pose estimation comprises the following steps:

[0022] Finding key frames with a common view by matching the current frame with all key frame images in the map According to the key frames with a common view Finding 3D map points with a common view Establishing the following minimum reprojection error expression as an objective function:

[0023]

[0024] In the formula: , are the rotation matrix and translation vector corresponding to the camera for shooting the i-th frame, i.e. the camera pose; is the i-th map point; is the number of common view key frames; is the number of common view map points; indicates whether the i-th map point has a projection on the j-th common view key frame, and if it has a projection , otherwise ; indicates a reprojection function that projects 3D point coordinates to a 2D plane; is the feature point coordinate corresponding to the map point on the key frame ; uses the camera pose corresponding to the last tracking frame or key frame , as initialization, and uses the Levenberg-Marquardt method to iteratively solve the objective function to obtain the pose.

[0025]

[0026] ​​​​As a preferred solution, in the step of registration, the optical flow method is used to combine the current tracking frame with the pose corresponding to the last tracking frame or key frame to calculate the pose of the current tracking frame, assuming that the brightness of the same object of the two frames of images to be estimated is constant, and the adjacent pixels have similar motion, and the Lucas-Kanade method provided by OpenCV is used to solve the point (x, y) x, y ) to obtain the matching point pair between the two frames of optical flow, the motion of the camera is solved by the PnP method R t ), and then the camera pose is obtained.

[0027] After processing the tracking frame, the value of Counter trk is set to 1, and the counting is restarted; if the change in the currently calculated pose exceeds the threshold λpos , the value of Counter trk is also set to 1, i.e. a normal frame is forced to be processed as a key frame.

[0028] As a preferred solution, the key frame is also used to construct a map and to correct a closed loop.

[0029] As a preferred solution, the pixel point p in the video frame satisfies any one of the stroke end point, the turning point and the three-way point of the text, and is selected as a candidate feature point.

[0030] As a preferred solution, the feature matching uses a brute force cross-validation matching method.

[0031] As a preferred solution, it further includes image blur detection for each tracking frame or key frame, if the image is blurred, the corresponding frame is processed as a normal frame, and the next frame of the corresponding frame is processed as a key frame.

[0032] An augmented reality tracking registration system for a text scene includes:

[0033] A frame division module is configured to input a video frame captured by a camera, and divide the current frame into a normal frame, a tracking frame and a key frame according to a frame decision rule.

[0034] A key frame tracking registration module is configured to perform image feature extraction, feature matching and pose estimation on the key frame to obtain a camera pose for tracking registration, and the image feature extraction detects stroke end points, turning points and three-way points of the text according to the following mathematical expressions:

[0035] The stroke end point satisfies:

[0036]

[0037] The stroke turning point satisfies:

[0038]

[0039] Stroke three fork points satisfy:

[0040]

[0041] Wherein, The sampling point set with similar gray values, The sampling point set with large gray value difference, Indicate the number of elements in the set Indicate the number of elements in the set , , , , The sampling point set adjacent and with similar gray values, The sampling point set adjacent and with large gray value difference, Indicate that the front and rear expressions satisfy simultaneously;

[0042] The tracking frame pose registration module is used for using the optical flow method to combine the current tracking frame with the pose corresponding to the last tracking frame or the key frame, to calculate the pose of the current tracking frame, and to register;

[0043] The normal frame pose linear change module is used for calculating the average change amount of each frame by the tracking frame and the key frame with known poses, the change amount of the two poses and the difference of the intermediate frame number, so that the pose of the normal frame is linearly changed from the previous pose to the next pose, and the augmented reality tracking registration of all video frames is completed.

[0044] A computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is executed by the processor to realize the augmented reality tracking registration method for the text scene.

[0045] Compared with the prior art, the present application has at least the following beneficial effects:

[0046] In the text scene, the existing AR tracking registration technology applied to the mobile terminal with light hardware resources has the problems of low accuracy and poor real-time performance in text tracking registration. Specifically, due to the high image texture repetition, large scale span and strong directionality in the text scene, the existing method is difficult to efficiently extract good feature points, which easily leads to low accuracy in subsequent tracking registration task. In addition, the existing tracking registration method based on parallel reconstruction also has the problem of poor real-time performance when applied to the mobile terminal. The present application divides the video frames shot by the camera into ordinary frames, tracking frames and key frames, and designs a text stroke detection method. The key frame can detect the stroke endpoints, turning points and three-way points of the text in the image feature extraction process, thereby improving the extraction capability of the text image features and achieving higher accuracy in the feature matching result. At the same time, the present application also designs a tracking strategy. For the tracking frame, the optical flow method is used to combine the tracking frame with the pose corresponding to the previous tracking frame or key frame, calculate the pose of the current tracking frame, and perform registration. For the ordinary frame, the tracking frame and the key frame are both known poses, and the average change amount of each frame is obtained by calculating the change amount of the two poses and the difference in the number of frames between them, so that the pose of the ordinary frame changes linearly from the previous pose to the next pose. Finally, the augmented reality tracking registration of all video frames is efficiently completed, the tracking registration processing time is shortened, and the real-time performance is improved. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and other related drawings can also be obtained by those skilled in the art without creating labor.

[0048] Figure 1 The text image feature extraction schematic diagram of the augmented reality tracking registration method for the text scene in the embodiment of the present application;

[0049] Figure 2 The flowchart of the augmented reality tracking registration method for the text scene in the embodiment of the present application;

[0050] Figure 3 The schematic diagram of the fuzzy image frame solution method in the embodiment of the present application;

[0051] Figure 4 The machine vision diagram of the key word scanning and tracking on the page of the textbook content in the embodiment of the present application;

[0052] Figure 5 The machine vision diagram of the knowledge point key word calibration on the textbook using the AR label in the embodiment of the present application;

[0053] Figure 6 The camera moves after the AR label of the embodiment of the application calibrates the key points of knowledge on the textbook by machine vision. DETAILED DESCRIPTION

[0054] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, but not all the embodiments of the application. Based on the embodiments in the application, other embodiments can be obtained by those skilled in the art without creative effort.

[0055] Please refer to Figure 2 The augmented reality tracking registration method for a text scene in the embodiment of the application includes the following steps.

[0056] S1, taking a video frame shot by a camera as input.

[0057] S2, dividing the current frame into three types: normal frame, tracking frame and key frame by a frame decision rule.

[0058] The specific rule is as follows:

[0059] (1) Key frame: the first frame or the key frame counter Counter key When equal to 0, the input frame is taken as a key frame; Counter key The initial value can be 9, that is, one key frame is taken every 10 frames, and the value can be given different values according to the device performance; every time a frame (including tracking frame and normal frame) is processed, Counter key Subtract one, in special cases Counter key It will be directly modified, that is, not fixed to take one key frame every 10 frames; after key frame processing occurs Counter key Reset to the initial value.

[0060] (2) Tracking frame: the counter Counter trk When equal to 0, the input frame is taken as a tracking frame; Counter trk The initial value is 1, and the value is reduced by one every time a normal frame is processed; when key frame or tracking frame processing occurs Counter trk Reset to 1.

[0061] (3) Normal frame: the rest of the frames are normal frames.

[0062] a. Keyframe processing method: the key frame is sequentially subjected to image feature extraction, feature matching, pose estimation to obtain camera pose for tracking registration, the key frame is also used to construct a map, loop correction. After key frame processing, the Counter key is set to 9, Counter trk is set to 1, and the counting is restarted.

[0063] The image feature extraction method is as follows:

[0064] As shown in Figure 1 , three rules are designed to detect the stroke end point, turning point, and three-way point of the text:

[0065] The stroke end point should satisfy rule 1:

[0066]

[0067] The stroke turning point should satisfy rule 2:

[0068]

[0069] The stroke three-way point should satisfy rule 3:

[0070]

[0071] wherein, is a set of sampling points with similar gray values, is a set of sampling points with large gray value differences, represents the number of elements in the set , represents the number of elements in the set , , , are sets of adjacent sampling points with similar gray values, are sets of adjacent sampling points with large gray value differences, indicates that the front and back expressions are satisfied at the same time; as long as one of the above three rules is satisfied, the pixel point p can be selected as a candidate feature point.

[0072] Feature matching: a brute force cross-validation matching method is used.

[0073] Pose estimation: first, the current frame is matched with all key frame images in the map to find key frames with a common view area. Then, the common view map points (3D) are calculated according to these common view key frames, and the following minimum re-projection error expression is established as the objective function:

[0074]

[0075] wherein: , are the rotation matrix and translation vector of the camera pose corresponding to the i-th key frame; are the rotation matrix and translation vector of the camera pose corresponding to the i-th key frame; are the rotation matrix and translation vector of the camera pose corresponding to the i-th key frame; is the i-th map point; is the number of co-visible key frames; is the number of co-visible map points; represents whether the i-th map point has a projection in the j-th co-visible key frame, and if it has a projection , otherwise ; represents a re-projection function that projects 3D point coordinates to a 2D plane; is the feature point coordinate corresponding to the map point on the key frame .

[0076] using the camera pose corresponding to the last tracking frame or key frame , as initialization, the pose is obtained by iteratively solving the objective function using the Levenberg-Marquardt method.

[0077] b. Tracking frame processing method: using the optical flow method, combined with the pose corresponding to the last tracking frame or key frame, the pose of the current tracking frame is calculated, and registration is performed. In the optical flow method, it is assumed that the brightness of the same object of the two frames of images to be estimated is constant, and the adjacent pixel points have similar motion, which is usually true in the text scene. The Lucas-Kanade method provided by OpenCV is used to solve the running of the point x, y ), and the matching point pair between the two frames of the optical flow is obtained, and the camera motion Rt ] is solved by PnP method, and then the camera pose is obtained; after processing the tracking frame, Counter trk is set to 1, and the count is reset; if the change amount of the current calculated pose exceeds the threshold λpos , then Counter trk is also set to 1, that is, a normal frame is forced to be processed as a key frame to prevent large errors caused by the failure of the optical flow method and to ensure the accuracy of the pose estimation.

[0078] c. Normal frame processing method: the normal frame is located between the tracking frame or the key frame, and the tracking frame and the key frame are both known poses, and the average change amount of each frame is obtained by calculating the change amount of the two poses and the difference in the number of frames, so that the pose of the normal frame is linearly changed from the previous pose to the next pose. ​​

[0079] S3, blurring image solution method

[0080] Serious blurred images can not extract useful features and noise can also cause false matches, and the poses of tracking frames and key frames are difficult to estimate accurately. First, image blur detection is needed for each tracking frame or key frame. If it is found that the image is too blurred, the frame is treated as a normal frame, and the next frame is treated as a key frame. The reason for setting it as a key frame is that image blur is usually caused by a large change in camera pose for a short time. A large change in pose will cause large pose errors or serious failures in the optical flow method of tracking frames. Therefore, the frame is directly treated as a key frame.

[0081] As shown in Figure 3 , the first frame is set as the target frame, and the first frame is also a key frame. The pose estimation method of step a is used to estimate the pose; the next frame (the second frame) is a normal frame, and the third frame is a tracking frame. The pose is calculated using the tracking frame processing method of step b, that is, the pose of the first frame is combined with the optical flow method to calculate the pose of the third frame. Then, the pose of the second frame is taken as the middle value of the change amount. The fifth frame is a tracking frame, and its pose is calculated using the optical flow method according to the pose of the third frame. Then, the fourth frame takes the middle value of the third and fifth frames as the pose. The seventh frame should be a tracking frame, but image blur detection finds that the image is too blurred, so the blurring image solution method of step S3 is used, and the frame is treated as a normal frame. The next frame (the eighth frame) is treated as a key frame. The same method is used to process the following frames.

[0082] Compared with the tracking registration method based on parallel reconstruction ORB-SLAM3 (reference: Campos C, Elvira R, Rodriguez J J G, et al. ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial, and Multimap SLAM[J]. IEEE Transactions on Robotics, 2021, 37(6): 1874-1890.). ORB-SLAM3: First, ORB-SLAM3 uses an ORB detector to detect key points in the input image and uses an ORB descriptor to extract features around these key points. Tracking module: Track these feature points in combination with the local map, then estimate the camera pose, which can further be registered in AR. Mapping module: Select the most informative feature image frame as the key frame, and calculate the 3D position of the feature points on the key frame as the map point, use these map points to construct the environment map, the map is represented as a set of key frames, use loop closure detection to correct the map, achieve high-precision positioning and mapping. The comparison results are as follows:

[0083]

[0084] The image sequence in the table is a video taken by simulating the user's textbook learning scene, a total of 6 videos of different textbook covers or content pages; RMSE The root mean square error is the Euclidean distance between the X coordinate of the bounding box vertex and the true coordinate , which can reflect the average accuracy of the registration method, unit: mm (millimeter), the calculation formula is as follows:

[0085]

[0086] Time per frame TPF , that is, the ratio of the time taken by the tracking registration method to process the image sequence to the number of images, unit: ms (millisecond), is an index for measuring the real-time performance of the method, , which is a measure of video smoothness. From the table, it can be seen that the method is 1 times faster than ORB-SLAM3, and the accuracy is improved by 29%.

[0087] Take scanning and tracking keywords on the page of the textbook content page as an example, as shown in Figure 4 Figure 4 To track and register the textbook page, click the scan textbook button to track and register. As shown in Figure 5 , use AR tags to calibrate the knowledge point keywords on the textbook. As shown in Figure 6 ​As shown, although the camera moves, the system can accurately calibrate the AR label to the relevant text. The effectiveness of the text scene-oriented augmented reality tracking registration method of the embodiment of the application is verified.

[0088] Another embodiment of the application also provides a text scene-oriented augmented reality tracking registration system, comprising:

[0089] a frame division module, configured to input a video frame captured by a camera, and divide the current frame into a normal frame, a tracking frame and a key frame according to a frame decision rule;

[0090] a key frame tracking registration module, configured to sequentially perform image feature extraction, feature matching and pose estimation on the key frame to obtain a camera pose for tracking registration, wherein the image feature extraction detects stroke endpoints, turning points and three-way points of the text according to the following mathematical expressions:

[0091] the stroke endpoints satisfy:

[0092]

[0093] the stroke turning points satisfy:

[0094]

[0095] the stroke three-way points satisfy:

[0096]

[0097] wherein, is a set of sampling points with similar gray values, is a set of sampling points with large gray value differences, denotes the number of elements in the set denotes the number of elements in the set , , , , are sets of adjacent sampling points with similar gray values, respectively, are sets of adjacent sampling points with large gray value differences, respectively, denotes that the front and rear expressions are satisfied simultaneously;

[0098] a tracking frame pose registration module, configured to use an optical flow method to combine the tracking frame with the pose corresponding to the last tracking frame or key frame, calculate the pose of the current tracking frame, and perform registration.

[0099] The common frame pose linear change module is configured to obtain the average change amount of each frame by calculating the average change amount of the two pose change amounts and the difference of the intermediate frame number between the tracking frame and the key frame with the known pose, so that the pose of the common frame is linearly changed from the previous pose to the next pose, and the augmented reality tracking registration of all video frames is completed.

[0100] Another embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the text scene-oriented augmented reality tracking registration method.

[0101] For example, the instructions stored in the memory can be divided into one or more modules / units, which are stored in the computer readable storage medium and executed by the processor to complete the text scene-oriented augmented reality tracking registration method of the present application. The one or more modules / units can be a series of computer readable instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the server.

[0102] The electronic device can be a smart phone, a notebook, a palm computer, a cloud server and other computing devices. The electronic device can include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the electronic device can further include more or less components, or combine certain components, or different components, for example, the electronic device can further include an input / output device, a network access device, a bus, etc.

[0103] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0104] The memory can be an internal storage unit of the server, such as a hard disk or a memory of the server. The memory can also be an external storage device of the server, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory can also include both the internal storage unit and the external storage device of the server. The memory is used to store the computer readable instructions and other programs and data required by the server. The memory can also be used to temporarily store data that has been output or will be output.

[0105] It should be noted that the information interaction and execution process between the above module units are based on the same concept as the method embodiments, and the specific functions and technical effects brought about can be referred to the method embodiments part, which will not be repeated here.

[0106] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above division of functional units and modules is taken as an example for illustration, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit or module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit. In addition, the specific name of each functional unit or module is only for convenient distinction, and does not limit the protection scope of the present application. The specific working process of the unit or module in the system can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0107] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the computer program is executed by a processor, the steps of the above-mentioned various method embodiments can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms. The computer readable medium at least includes any entity or device capable of carrying the computer program code to the photographing device / terminal equipment, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium. For example, U disk, mobile hard disk, magnetic disk or optical disk, etc.

[0108] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in a certain embodiment can be referred to the relevant description of other embodiments.

[0109] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. An augmented reality tracking and registration method for text-based scenes, characterized in that, Includes the following steps: The input video frame captured by the camera is divided into normal frame, tracking frame, and key frame by the frame decision rule; The keyframes are sequentially subjected to image feature extraction, feature matching, and pose estimation to obtain the camera pose for tracking and registration. The image feature extraction detects the stroke endpoints, turning points, and bifurcation points of the characters according to the following mathematical expressions: Stroke endpoints satisfy: The stroke turning point satisfies: The three-way intersection of the strokes satisfies: in, It is a set of sampling points with similar gray values. It is a set of sampling points with large differences in grayscale values. Represents a set The number of elements in the middle, Represents a set The number of elements in the middle, , , These are sets of neighboring sampling points with similar gray values. These are sets of neighboring sampling points with large differences in grayscale values. This indicates that both the preceding and following expressions satisfy the condition. Using optical flow, the pose of the current tracking frame is calculated by combining the tracking frame with the pose corresponding to the previous tracking frame or key frame, and then registered. By using tracking frames and keyframes with known poses, the average change in pose between two consecutive frames is calculated, along with the difference in the number of intermediate frames. This allows the pose of ordinary frames to change linearly from the previous pose to the next pose, thus completing the augmented reality tracking registration for all video frames.

2. The augmented reality tracking and registration method for text-based scenes according to claim 1, characterized in that, In the step of dividing the current frame into normal frames, tracking frames, and key frames using frame decision rules: First frame or keyframe counter Counter key When the input value is 0, the frame is the keyframe; this is achieved through initialization. Counter key The value refers to the number of keyframes selected, with different values ​​assigned based on device performance; each frame processed includes tracking frames and normal frames. Counter key Subtract one, in special circumstances Counter key It will be directly modified, meaning it's not a fixed set number of frames per keyframe; after keyframe processing occurs... Counter key Reset to initial value; counter Counter trk The frame input when the value is 0 is used as the tracking frame; Counter trk The initial value is 1, and it decreases by one for each normal frame processed; after a keyframe or tracking frame is processed, Counter trk Reset to 1; Apart from keyframes and tracking frames, all other frames are ordinary frames.

3. The augmented reality tracking and registration method for text-based scenes according to claim 2, characterized in that, The pose estimation includes the following steps: Keyframes with shared view areas are identified by matching the current frame with all keyframe images in the map. Based on keyframes with shared view areas Find the common 3D map points The objective function is to establish the following expression for the minimum reprojection error: In the formula: , The shooting of the first The rotation matrix and translation vector corresponding to the frame camera, i.e., the camera pose; For the first One map point; This refers to the number of shared keyframes. The number of map points that are in common view; Indicates the first The map point at the ... Does each shared keyframe have projection? If so, then... ,otherwise ; This represents the reprojection function, which projects the coordinates of a 3D point onto a 2D plane. For map points In keyframe The coordinates of the corresponding feature point above; Use the camera pose corresponding to the previous tracking frame or keyframe. , As initialization, the pose is obtained by iteratively solving the objective function using the Levenberg–Marquardt method.

4. The augmented reality tracking and registration method for text-based scenes according to claim 3, characterized in that, In the step of using optical flow to calculate the pose of the current tracking frame by combining the tracking frame with the pose corresponding to the previous tracking frame or keyframe and registering it, it is assumed that the brightness of the same object in the two frames of images to be estimated remains unchanged, and adjacent pixels have similar motion. The point is solved using the Lucas-Kanade method provided by OpenCV. x, y The operation of the optical flow is used to obtain the matching point pairs between two frames, and the camera motion is solved using the PnP method. R t ], thereby obtaining the camera pose; After tracking frame processing, Counter trk Set to 1 and start counting again; if the currently calculated pose change exceeds the threshold. λpos Then Counter trk It is also set to 1, which means that a keyframe processing is forced after a normal frame.

5. The augmented reality tracking and registration method for text-based scenes according to claim 1, characterized in that, The keyframes are also used for map construction and loop closure correction.

6. The augmented reality tracking and registration method for text-based scenes according to claim 1, characterized in that, A pixel p in the video frame is selected as a candidate feature point if it satisfies any one of the stroke endpoints, turning points, and bifurcations of the character.

7. The augmented reality tracking and registration method for text-based scenes according to claim 1, characterized in that, The feature matching uses a brute-force cross-validation matching method.

8. The augmented reality tracking and registration method for text-based scenes according to claim 1, characterized in that, It also includes performing image blur detection on each tracking frame or keyframe. If the image is blurry, the corresponding frame is treated as a normal frame, and the next frame is treated as a keyframe.

9. An augmented reality tracking and registration system for text-based scenes, characterized in that, include: The frame segmentation module is used to input video frames captured by the camera and, through frame decision rules, divides the current frame into normal frames, tracking frames, and key frames. The keyframe tracking registration module is used to sequentially perform image feature extraction, feature matching, and pose estimation on keyframes to obtain the camera pose for tracking registration. The image feature extraction detects the stroke endpoints, turning points, and bifurcation points of characters according to the following mathematical expressions: Stroke endpoints satisfy: The stroke turning point satisfies: The three-way intersection of the strokes satisfies: in, It is a set of sampling points with similar gray values. It is a set of sampling points with large differences in grayscale values. Represents a set The number of elements in the middle, Represents a set The number of elements in the middle, , , These are sets of neighboring sampling points with similar gray values. These are sets of neighboring sampling points with large differences in grayscale values. This indicates that both the preceding and following expressions satisfy the condition. The tracking frame pose registration module is used to calculate the pose of the current tracking frame by combining the pose of the tracking frame with that of the previous tracking frame or key frame using the optical flow method, and then register it. The normal frame pose linear transformation module is used to calculate the average change of each frame by taking the known pose of the tracking frame and key frame, and the difference between the pose changes of the two consecutive frames and the number of intermediate frames, so that the pose of the normal frame changes linearly from the previous pose to the next pose, thus completing the augmented reality tracking registration of all video frames.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the augmented reality tracking and registration method for text-oriented scenes as described in any one of claims 1 to 8.

Images