Image recognition-based electromagnetic point reading pen positioning method

Abstract

The application discloses an electromagnetic point reading pen positioning method based on image recognition, and comprises the following steps: S1, collecting a textbook page image, combining an improved SuperPoint model and a time sequence processing unit to complete feature extraction, alignment, difference calculation and learnable parameter optimization of a key point set and a descriptor vector; S2, generating a layout template; S3, extracting and screening the key point set to generate current frame feature data containing two-dimensional coordinates, confidence and a normalized descriptor; S4, generating visual registration parameters; S5, combining the visual registration result to compare and update initial electromagnetic prior registration parameters, and outputting effective registration parameters of a current operation cycle; and S6, generating a positioning coordinate sequence and updating visual registration prior parameters. The application can realize high-precision and low-drift page positioning under different paper materials, illumination conditions and long-time use conditions, and is suitable for intelligent point reading, interactive teaching and high-precision paper surface interaction scenes.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction and intelligent sensing technology, and in particular to a method for locating an electromagnetic reading pen based on image recognition. Background Technology

[0002] Existing pen positioning technologies primarily rely on either electromagnetic induction or optical image recognition to determine the pen tip's coordinates on the page. Electromagnetic induction positioning technology uses an electromagnetic grid arranged on the page or writing board to calculate the pen tip's position in the device's coordinate system based on the electromagnetic signals emitted or received by the pen tip. It offers advantages such as high real-time performance and independence from external lighting conditions. However, the accuracy of electromagnetic sensors is affected by hardware resolution, transmission delay, and environmental electromagnetic interference, leading to random noise and systematic biases in the positioning results, particularly noticeable coordinate drift in edge regions. Optical image recognition positioning technology, on the other hand, relies on a camera to capture page images and calculates the pen tip's position through feature extraction and matching. While it offers high positioning accuracy under rich page structure features and good lighting conditions, its stability decreases significantly under varying lighting, occlusion, or blurring conditions. Furthermore, the camera's sampling frequency and processing latency limit its real-time performance.

[0003] To improve positioning accuracy, some existing solutions attempt to combine electromagnetic induction positioning with optical recognition, compensating for errors in a single mode by correcting registration parameters at specific times. These fusion schemes mostly use fixed weights or simple weighting for data fusion, failing to dynamically adjust the fusion strategy based on the matching quality of visual registration and the noise level of the electromagnetic signal during actual operation. Current technologies often rely solely on single-frame matching for visual registration, lacking utilization of temporal continuity, leading to decreased registration accuracy when the pen tip moves rapidly or when some page features are missing. The update mechanisms for electromagnetic and visual registration parameters are mostly periodic refreshes rather than dynamic optimization based on real-time detection results, which can easily lead to cumulative errors when multiple interference factors are superimposed.

[0004] Existing technologies for e-reader pen positioning suffer from several drawbacks, including a lack of adaptability in fusion strategies, untimely updates to registration parameters, and insufficient utilization of temporal information. These limitations make it difficult to simultaneously achieve both accuracy and real-time performance in complex environments. Consequently, there is still significant room for improvement in the positioning accuracy and stability of e-readers in applications such as education and interactive publications. Summary of the Invention

[0005] One objective of this invention is to propose an image recognition-based electromagnetic reading pen positioning method. This invention fully utilizes the improved SuperPoint feature extraction algorithm, temporal feature processing mechanism, and visual and electromagnetic data fusion registration technology. It describes in detail the processing flow for achieving high-precision real-time positioning of the reading pen in the context of textbook pages, and has the advantages of high positioning accuracy, strong anti-interference ability, and fast response speed.

[0006] An image recognition-based electromagnetic reading pen positioning method according to an embodiment of the present invention includes the following steps: S1. Collect images of the textbook pages and perform preprocessing sequentially. Combine the improved SuperPoint model and time-series processing unit to complete feature extraction, alignment, difference calculation and learnable parameter optimization of the key point set and descriptor vector. S2. Extract the set of key points from the target textbook page image, calculate the homography geometric transformation matrix by combining the reference points of the page logical coordinate system, complete the coordinate transformation and generate the page template; S3. Collect and correct the current page image frame, extract and filter the key point set, and generate current frame feature data containing two-dimensional coordinates, confidence and normalized descriptors; S4. Based on the matching of the layout template and the feature data of the current frame, establish a correspondence, use the random sampling consistency algorithm to estimate and refine the homography geometric transformation matrix, and generate visual registration parameters; S5. Combine the visual registration results with the comparison and update the initial electromagnetic prior registration parameters, and output the effective registration parameters for the current running cycle. S6. Generate the positioning coordinate sequence and update the visual registration prior parameters.

[0007] Optionally, the improved SuperPoint model includes the following modules: The system includes an input preprocessing module, a shared feature encoder with temporal consistency constraints, a keypoint detection branch, a descriptor extraction branch, an upsampling and mesh reconstruction module, a learnable parameter-driven keypoint selection module, and a training control module that integrates physical enhancement and temporal consistency loss.

[0008] Optionally, the feature extraction process includes: The input image is fed into a shared encoder and generated as an initial two-dimensional feature map through the first convolutional layer. Batch normalization is applied at the output of each convolutional layer to stabilize the feature distribution, and a non-linear activation function is applied to enhance the feature representation capability. During the stacking of convolutional layers, the spatial resolution of the initial two-dimensional feature map is gradually reduced and the channel dimension is increased through downsampling operations, thereby encoding multi-scale layout structure features; Multi-scale page layout features refer to the set of feature representations that simultaneously encode the overall geometric layout information and local texture information of a page at different spatial resolutions through layer-by-layer convolution and downsampling operations in a convolutional neural network.

[0009] The shared encoder outputs a two-dimensional feature map containing spatial location information and local texture information at its end. In a shared encoder, convolutional kernels perform weighted summation and nonlinear mapping within the local receptive field of an image, ensuring that the response value of each feature map pixel is correlated with the pixel distribution pattern of that location and its neighborhood in the input image, thus preserving spatial location information. Simultaneously, the kernel weights learn sensitivity to different local grayscale changes, edge contours, and texture structures through backpropagation during training, enabling the output feature map to represent the local texture information of the page. Through multiple layers of convolution and downsampling operations, the responses of convolutional kernels at different scales are gradually fused, encoding multi-level information from global layout to local texture details in the final two-dimensional feature map. Therefore, this two-dimensional feature map naturally contains both spatial location and local texture information and can be directly used by the keypoint detection branch and descriptor extraction branch.

[0010] The two-dimensional feature map output by the shared encoder is input into the key point detection branch and the descriptor extraction branch respectively to generate a key point probability heatmap and a set of descriptor vectors at the corresponding positions. The keypoint probability heatmap is filtered, and the set of keypoints and their coordinates in the image coordinate system are output. Descriptor vectors are extracted at the corresponding positions.

[0011] Optionally, S1 specifically includes: S11. Collect images of textbook pages, the materials of which include ordinary offset paper and laminated cardboard, covering printing production batches with different color calibration parameters, different ink ratios, and different page registration accuracies; S12. Scale the textbook page image to a uniform resolution and convert it to a grayscale image; S13. For each grayscale image, randomly select four vertices of the grayscale image and make pixel offsets in the horizontal and vertical directions respectively not exceeding three percent of the page width and height. Calculate a 3×3 homography geometric transformation matrix based on the vertex coordinates after pixel offset and the original vertex coordinates. Perform homography transformation on the grayscale image according to the homography geometric transformation matrix to generate the transformed page image. S14. Perform physical enhancement processing on the page image to generate enhanced image pairs; the physical enhancement processing includes: applying paper warping distortion, superimposing local bright spots, superimposing random mesh patterns, adjusting the brightness and saturation of color channels, adding Gaussian noise, adding compression artifacts, superimposing linear illuminance gradients, and superimposing radial illuminance gradients. S15. Input the enhanced image to the improved SuperPoint model and extract the key point set and corresponding descriptor vectors according to the feature extraction process; S16. For one page image in the enhanced image pair, perform geometric alignment of the key point probability heatmap of the other page image according to the homography geometric transformation matrix, and calculate the probability difference at the alignment position. S17. In the enhanced image pair, extract the descriptor vectors for the corresponding positions after geometric alignment, calculate the Euclidean distance between the positive sample pair and the negative sample pair, minimize the distance of the positive sample pair and maximize the distance of the negative sample pair. S18. In manually annotated textbook page samples, real keypoint labels are used to guide the classification training of keypoint probability heatmaps. The keypoint detection probability threshold and non-maximum suppression radius are set as learnable parameters and iteratively updated along with the weights of the improved SuperPoint model until convergence, thus obtaining the optimized learnable parameters. Among them, the keypoint detection probability is generated by the keypoint detection branch of the improved SuperPoint model, representing the probability value of each pixel position belonging to the keypoint category. Two-dimensional pixel coordinates and validity labels of stable feature points in each page image of the textbook page sample were manually annotated; S19. A temporal processing unit is introduced between the shared encoder and the key point detection branch and the descriptor extraction branch. The temporal processing unit includes an optical flow estimation module, a cross-frame feature mapping module and a difference calculation module. The continuously acquired video frames of the textbook pages are sequentially input into the shared encoder of the improved SuperPoint model to obtain two-dimensional feature maps of adjacent frames; The pixel-level optical flow field between adjacent frames is calculated based on the optical flow estimation module. The key point probability heatmap and descriptor vector set of the previous frame are mapped to the corresponding spatial location of the next frame through the cross-frame feature mapping module. In the difference calculation module, the difference between the keypoint detection probability after mapping and the keypoint detection probability at the corresponding position in the next frame is calculated position by position to obtain the keypoint probability difference map; the Euclidean distance between the mapped descriptor vector and the descriptor vector at the corresponding position in the next frame is calculated to obtain the descriptor difference matrix; the keypoint probability difference map and the descriptor difference matrix are used as temporal consistency constraints and participate in the optimization of the improved SuperPoint model together with the single-frame training loss.

[0012] Optionally, S2 specifically includes: S21. Scale the target textbook page image to a uniform resolution and convert it to a grayscale image; S22. Input the grayscale image into the improved SuperPoint model, and extract the key point set and corresponding descriptor vector according to the feature extraction process; S23. The optimized learnable parameters are used to filter the key point probability heatmap to obtain a set of key points and their coordinates in the image coordinate system, and descriptor vectors are extracted at the corresponding positions. The screening process involves two steps: Iterate through each pixel position in the keypoint probability heatmap and retain only candidate positions whose keypoint detection probability is greater than or equal to the keypoint detection probability threshold. Using the non-maximum suppression radius as the neighborhood range, only the candidate positions with the highest key point detection probability are retained in each neighborhood, and other candidate positions are eliminated to obtain the final set of key points and the coordinates of the key point set in the image coordinate system. Descriptor vectors are extracted at the corresponding positions. S24. Determine the reference point of the page logical coordinate system of the target textbook page, and calculate the homography geometric transformation matrix from the image coordinate system to the page logical coordinate system. S25. Based on the homography geometric transformation matrix, transform the image coordinates of the key point set into page logical coordinates to obtain the feature point set in the page logical coordinate system; S26. Assign an index identifier to each feature point in the feature point set and establish a positional correspondence with the page content index to generate a layout template. The layout template includes page logical coordinate system parameters, a feature point set, an index identifier for each feature point, and its descriptor vector.

[0013] Optionally, S3 specifically includes: S31. Load the camera intrinsic parameter matrix and distortion coefficient of the built-in camera of the reading pen, and set the output resolution and timestamp recording parameters; S32. Acquire the current page image frame and record the timestamp to obtain the original page image; S33. Perform distortion correction processing on the original page image based on the parameters of Zhang Zhengyou calibration method, and perform brightness normalization by adaptive histogram equalization to obtain the corrected page image. S34. Input the corrected page image into the improved SuperPoint model, extract the key point set and corresponding descriptor vector according to the feature extraction process, and update the timing state in combination with the timing processing unit. S35. Use the optimized keypoint detection probability threshold and non-maximum suppression radius to filter the keypoint set of the current frame, perform keypoint probability heatmap filtering operation on the current frame, and obtain the two-dimensional coordinates of the keypoint set in the image coordinate system and the corresponding confidence level.

[0014] S36. Extract descriptor vectors from the key point set locations, and normalize each descriptor vector according to its vector magnitude to obtain normalized descriptors. Output the two-dimensional coordinates of the key points in the current frame in the image coordinate system, the corresponding confidence score, and the normalized descriptor.

[0015] Optionally, S4 specifically includes: S41. Read the layout template and the two-dimensional coordinates and normalized descriptors of the key points in the image coordinate system; S42. Perform nearest neighbor retrieval and matching pair filtering within the descriptor metric space to obtain a candidate matching pair set; The descriptor metric space refers to a high-dimensional vector space composed of all normalized descriptors, where the similarity between any two descriptor vectors can be calculated based on a preset Euclidean distance, which is used to measure the degree of matching between descriptors.

[0016] S43. Establish the correspondence between the current frame image coordinates and the page logical coordinates based on the candidate matching pair set; S44. Based on the correspondence, the random sampling consistency algorithm is used to estimate the online homography geometric transformation matrix from the image coordinate system to the page logical coordinate system, and the parameters are refined according to the interior point set. The inner point set is the set of matching point pairs whose error is determined to be less than or equal to a preset threshold during the online estimation of homography geometric transformation matrix. S45. Record the refined online estimated homography geometric transformation matrix as visual registration parameters and synchronize it to the temporal processing unit to update the visual registration prior parameters. Based on the key point probability difference map output by the temporal processing unit, calculate the proportion of high difference regions. If it exceeds the set difference threshold, apply confidence weight adjustment to the current visual registration parameters and use the adjustment result as a weight factor to synchronously adjust the update amplitude of the electromagnetic prior registration parameters.

[0017] The specific steps for calculating the proportion of the high-difference region include: First, perform pixel-level traversal on the keypoint probability difference map, set a difference threshold, filter out pixel positions with difference values ​​greater than or equal to the difference threshold, and count their number as the number of high-difference region pixels; then divide this number by the total number of pixels in the image to obtain the proportion of high-difference regions; this proportion reflects the degree of significant change in the keypoint detection probability distribution between the current frame and the previous frame.

[0018] The application of confidence weight adjustment includes: Based on the keypoint probability difference map output by the temporal processing unit, the number of pixel regions in which the keypoint detection probability changes exceed a preset threshold is counted, and the proportion of such regions in the entire image is calculated as an indicator of the proportion of high difference regions.

[0019] The proportion of this high-difference region is mapped to a confidence weight factor to measure the reliability of the current visual registration parameters.

[0020] If the confidence weight factor is lower than the set threshold, the current visual registration is considered unstable. The current visual registration parameters are then reduced and adjusted, and the adjusted registration parameters are taken as the effective visual registration result in the current running cycle.

[0021] The update magnitude of the synchronously adjusted electromagnetic prior registration parameters: While updating the visual registration parameters, a confidence weighting factor generated based on the probability difference map of key points in the current frame determines the update magnitude of the electromagnetic prior registration parameters. Specifically, when the visual registration confidence is high, it indicates that the image key point matching is stable, allowing for a larger adjustment of the electromagnetic prior registration parameters, thereby accelerating the convergence of visual-electromagnetic registration fusion. Conversely, when the visual registration confidence is low, it indicates uncertainty in image matching. To avoid visual errors introducing electromagnetic registration drift, the update magnitude of the electromagnetic prior registration parameters is reduced proportionally, and only small-step correction operations are performed.

[0022] Optionally, S5 specifically includes: S51. Multi-channel electromagnetic signals are collected by an electromagnetic induction grid and corresponding timestamps are generated. The electromagnetic induction grid is a two-dimensional induction coil array structure deployed inside the interactive medium, used to sense the high-frequency electromagnetic signals emitted by the pen tip when it approaches or is touched; the signal is converted into a multi-channel electromagnetic signal by the built-in acquisition module and timestamped for subsequent electromagnetic positioning calculations.

[0023] S52. Preprocess the multi-channel electromagnetic signal and calculate the original electromagnetic coordinates in the device coordinate system; The preprocessing includes: performing bandpass filtering on the sensing signals of each channel to suppress environmental electromagnetic interference and high-frequency noise; performing synchronization alignment based on the sampling clock to ensure the consistency of data time for each channel; normalizing the amplitude of the synchronized signal to eliminate the sensitivity differences between the sensing coils; and, after the preprocessing is completed, using a coordinate inversion algorithm to calculate the original electromagnetic coordinates of the reading pen in the device coordinate system based on the amplitude distribution and phase difference of the sensing signals of each channel.

[0024] S53. During the session initialization phase, calculate the initial electromagnetic prior registration parameters from the device coordinate system to the page logical coordinate system based on the known page logical coordinate system reference point. Page logical coordinate system reference points refer to a set of predefined and fixed-position calibration points in the page logical coordinate system, used to establish the geometric mapping relationship between the page logical coordinate system and the device coordinate system and image coordinate system.

[0025] The reference point of the page logical coordinate system can be selected from the set of feature points in the page layout template. It is located at the four corners, edges or other stable printing mark positions that do not change with the page batch. Its two-dimensional coordinates in the page logical coordinate system are known constants and can be used as target coordinates in the registration calculation.

[0026] S54. Apply the online estimated homography geometric transformation matrix to the coordinates of the key point set obtained by the improved SuperPoint model in the current frame, and calculate the position of the key points in the page logical coordinate system; The positions of key points after visual registration and transformation are compared with the positions of the original electromagnetic coordinates under the corresponding timestamp in the page logical coordinate system, and the coordinate difference vector in the page logical coordinate system is calculated. Based on the coordinate difference vector, the registration parameter update operation is performed to adjust the translation, rotation and scale components in the initial electromagnetic prior registration parameters; S55. After the update is completed, the adjusted electromagnetic prior registration parameters are used as the valid registration parameters for the current running cycle and recorded in the registration parameter cache for real-time mapping of electromagnetic coordinates.

[0027] Optionally, S6 specifically includes: S61. Read the visual registration parameters and the statistics of the matching inlier rate and reprojection error of the visual registration parameters; read the electromagnetic original coordinates, timestamp, and electromagnetic prior registration parameters. S62. Statistically analyze the original electromagnetic coordinates within the sliding time window and estimate the electromagnetic noise variance using the sample variance calculation method. S63. Calculate the set of weight coefficients based on the visual matching inlier rate, reprojection error variance and electromagnetic noise variance. S64. Solve for the fusion registration parameters by combining the visual registration parameters and the electromagnetic prior registration parameters according to the set of weight coefficients. S65. Convert the original electromagnetic coordinates into page logical coordinates using the fusion registration parameters to generate a positioning coordinate sequence corresponding to the timestamp; S66. Output the final positioning coordinate sequence and fusion registration parameters, and synchronize the fusion registration parameters to the timing processing unit as visual registration prior parameters.

[0028] The beneficial effects of this invention are: This invention introduces an improved SuperPoint model and a shared encoder structure to simultaneously preserve the spatial location and local texture information of the page during feature extraction. By combining a learnable keypoint detection probability threshold and a non-maximum suppression radius, it achieves stable keypoint detection and descriptor extraction under varying lighting, viewing angles, and page texture interference, effectively improving the robustness and accuracy of feature point matching. Furthermore, this invention integrates temporal processing mechanisms such as optical flow estimation, cross-frame feature mapping, and difference calculation in the feature extraction stage. This ensures that the reading pen maintains continuous and consistent feature tracking even during rapid movement, partial occlusion, or low frame rate acquisition, thereby reducing the probability of positioning loss and jumps.

[0029] This invention introduces a fusion registration method of visual and electromagnetic dual-mode data in the positioning and registration stage. By adaptively calculating the weight coefficients through matching interior point rate, reprojection error variance and electromagnetic noise variance, and dynamically updating the fusion registration parameters, the final positioning accuracy and stability in the page logical coordinate system are significantly improved, effectively overcoming the defects of single visual or electromagnetic schemes being easily affected by environmental noise and attitude changes. Attached Figure Description

[0030] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is an overall flowchart of an electromagnetic reading pen positioning method based on image recognition proposed in this invention; Figure 2 This is a schematic diagram of the improved SuperPoint model structure of an electromagnetic reading pen positioning method based on image recognition proposed in this invention. Figure 3 This is a schematic diagram of the key point detection and screening process of an electromagnetic reading pen positioning method based on image recognition proposed in this invention. Figure 4 This is a schematic diagram of the feature extraction process of an electromagnetic reading pen positioning method based on image recognition proposed in this invention. Detailed Implementation

[0031] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0032] refer to Figure 1-4 An image recognition-based method for locating an electromagnetic reading pen includes the following steps: S1. Collect images of the textbook pages and perform preprocessing sequentially. Combine the improved SuperPoint model and time-series processing unit to complete feature extraction, alignment, difference calculation and learnable parameter optimization of the key point set and descriptor vector. S2. Extract the set of key points from the target textbook page image, calculate the homography geometric transformation matrix by combining the reference points of the page logical coordinate system, complete the coordinate transformation and generate the page template; S3. Collect and correct the current page image frame, extract and filter the key point set, and generate current frame feature data containing two-dimensional coordinates, confidence and normalized descriptors; S4. Based on the matching of the layout template and the feature data of the current frame, establish a correspondence, use the random sampling consistency algorithm to estimate and refine the homography geometric transformation matrix, and generate visual registration parameters; S5. Combine the visual registration results with the comparison and update the initial electromagnetic prior registration parameters, and output the effective registration parameters for the current running cycle. S6. Generate the positioning coordinate sequence and update the visual registration prior parameters.

[0033] This invention proposes a visual and electromagnetic fusion registration and positioning method based on an improved SuperPoint model and a temporal processing unit. Through the acquisition, preprocessing, and key point feature extraction of textbook page images, it achieves high-precision alignment of the key point set and descriptor vectors, and optimizes learnable parameters. By combining the page's logical coordinate system reference points to calculate the homography geometric transformation matrix, coordinate transformation is completed and a page template is generated, effectively ensuring the stability of page matching. During operation, the current page image frame is acquired and corrected in real time, extracting feature data containing two-dimensional coordinates, confidence levels, and normalized descriptors, and matching it with the page template. The homography geometric transformation matrix is ​​estimated and refined using a random sampling consistency algorithm to generate highly reliable visual registration parameters. Furthermore, the initial electromagnetic prior registration parameters are dynamically updated based on the visual registration results, achieving complementary advantages between visual and electromagnetic information. Finally, a high-precision positioning coordinate sequence is generated, and the registration prior parameters are continuously optimized, significantly improving the accuracy, stability, and real-time response capability of the positioning system.

[0034] In this embodiment, the improved SuperPoint model includes the following modules: The system includes an input preprocessing module, a shared feature encoder with temporal consistency constraints, a keypoint detection branch, a descriptor extraction branch, an upsampling and mesh reconstruction module, a learnable parameter-driven keypoint selection module, and a training control module that integrates physical enhancement and temporal consistency loss.

[0035] In this embodiment, the feature extraction process includes: The input image is fed into a shared encoder and generated as an initial two-dimensional feature map through the first convolutional layer. Batch normalization is applied at the output of each convolutional layer to stabilize the feature distribution, and a non-linear activation function is applied to enhance the feature representation capability. During the stacking of convolutional layers, the spatial resolution of the initial two-dimensional feature map is gradually reduced and the channel dimension is increased through downsampling operations, thereby encoding multi-scale layout structure features; Multi-scale page layout features refer to the set of feature representations that simultaneously encode the overall geometric layout information and local texture information of a page at different spatial resolutions through layer-by-layer convolution and downsampling operations in a convolutional neural network.

[0036] The shared encoder outputs a two-dimensional feature map containing spatial location information and local texture information at its end. In a shared encoder, convolutional kernels perform weighted summation and nonlinear mapping within the local receptive field of an image, ensuring that the response value of each feature map pixel is correlated with the pixel distribution pattern of that location and its neighborhood in the input image, thus preserving spatial location information. Simultaneously, the kernel weights learn sensitivity to different local grayscale changes, edge contours, and texture structures through backpropagation during training, enabling the output feature map to represent the local texture information of the page. Through multiple layers of convolution and downsampling operations, the responses of convolutional kernels at different scales are gradually fused, encoding multi-level information from global layout to local texture details in the final two-dimensional feature map. Therefore, this two-dimensional feature map naturally contains both spatial location and local texture information and can be directly used by the keypoint detection branch and descriptor extraction branch.

[0037] The two-dimensional feature map output by the shared encoder is input into the key point detection branch and the descriptor extraction branch respectively to generate a key point probability heatmap and a set of descriptor vectors at the corresponding positions. The keypoint probability heatmap is filtered, and the set of keypoints and their coordinates in the image coordinate system are output. Descriptor vectors are extracted at the corresponding positions.

[0038] In this embodiment, the generation of the keypoint probability heatmap is accomplished based on the keypoint detection branch of the improved SuperPoint model: The two-dimensional feature map output by the shared encoder is input into the key point detection branch, and local salient features are extracted through multi-layer convolution operations; Each pixel location is mapped to a keypoint detection probability distribution that belongs to either a "keypoint" or a "non-keypoint" through a specific probability mapping layer; The keypoint detection probabilities of all pixel locations are combined according to their spatial positions in the feature map to form a two-dimensional matrix, namely the keypoint probability heatmap, which is used to represent the confidence distribution of each location as a keypoint in the entire image.

[0039] The process of generating the set of descriptor vectors for the corresponding positions: The 2D feature map, containing spatial location and local texture information, output from the shared encoder, is input into the descriptor extraction branch. Within this branch, convolution operations, batch normalization, and nonlinear activation are performed sequentially to progressively aggregate texture and structural features from the local neighborhood. At the end of the convolution stack, a multi-channel feature representation is generated, and a fixed-length feature vector is extracted at each spatial location, forming a descriptor vector set that corresponds one-to-one with the spatial location of the 2D feature map. This descriptor vector set is used to characterize the local feature distribution at each spatial location for subsequent feature matching and cross-frame mapping.

[0040] In this embodiment, S1 specifically includes: S11. Collect images of textbook pages, the materials of which include ordinary offset paper and laminated cardboard, covering printing production batches with different color calibration parameters, different ink ratios, and different page registration accuracies; S12. Scale the textbook page image to a uniform resolution and convert it to a grayscale image; S13. For each grayscale image, randomly select four vertices of the grayscale image and make pixel offsets in the horizontal and vertical directions respectively not exceeding three percent of the page width and height. Calculate a 3×3 homography geometric transformation matrix based on the vertex coordinates after pixel offset and the original vertex coordinates. Perform homography transformation on the grayscale image according to the homography geometric transformation matrix to generate the transformed page image. S14. Perform physical enhancement processing on the page image to generate enhanced image pairs; the physical enhancement processing includes: applying paper warping distortion, superimposing local bright spots, superimposing random mesh patterns, adjusting the brightness and saturation of color channels, adding Gaussian noise, adding compression artifacts, superimposing linear illuminance gradients, and superimposing radial illuminance gradients. S15. Input the enhanced image to the improved SuperPoint model and extract the key point set and corresponding descriptor vectors according to the feature extraction process; S16. For one page image in the enhanced image pair, perform geometric alignment of the key point probability heatmap of the other page image according to the homography geometric transformation matrix, and calculate the probability difference at the alignment position. S17. In the enhanced image pair, extract the descriptor vectors for the corresponding positions after geometric alignment, calculate the Euclidean distance between the positive sample pair and the negative sample pair, minimize the distance of the positive sample pair and maximize the distance of the negative sample pair. S18. In manually annotated textbook page samples, real keypoint labels are used to guide the classification training of keypoint probability heatmaps. The keypoint detection probability threshold and non-maximum suppression radius are set as learnable parameters and iteratively updated along with the weights of the improved SuperPoint model until convergence, thus obtaining the optimized learnable parameters. Among them, the keypoint detection probability is generated by the keypoint detection branch of the improved SuperPoint model, representing the probability value of each pixel position belonging to the keypoint category. Two-dimensional pixel coordinates and validity labels of stable feature points in each page image of the textbook page sample were manually annotated; S19. A temporal processing unit is introduced between the shared encoder and the key point detection branch and the descriptor extraction branch. The temporal processing unit includes an optical flow estimation module, a cross-frame feature mapping module and a difference calculation module. The continuously acquired video frames of the textbook pages are sequentially input into the shared encoder of the improved SuperPoint model to obtain two-dimensional feature maps of adjacent frames; The pixel-level optical flow field between adjacent frames is calculated based on the optical flow estimation module. The key point probability heatmap and descriptor vector set of the previous frame are mapped to the corresponding spatial location of the next frame through the cross-frame feature mapping module. In the difference calculation module, the difference between the keypoint detection probability after mapping and the keypoint detection probability at the corresponding position in the next frame is calculated position by position to obtain the keypoint probability difference map; the Euclidean distance between the mapped descriptor vector and the descriptor vector at the corresponding position in the next frame is calculated to obtain the descriptor difference matrix; the keypoint probability difference map and the descriptor difference matrix are used as temporal consistency constraints and participate in the optimization of the improved SuperPoint model together with the single-frame training loss.

[0041] This invention proposes a textbook page feature extraction and enhancement training method combining an improved SuperPoint model and a temporal processing unit. It collects multiple batches of textbook page images with varying paper materials, color calibration parameters, ink ratios, and page registration accuracy. The method performs uniform resolution scaling, grayscale conversion, and controlled random pixel shift homography transformation. Combined with physical enhancement processes such as paper warping, bright spot overlay, halftone overlay, brightness and saturation adjustment, Gaussian noise reduction, compression artifact removal, and illumination gradient, it generates diverse training sample pairs. Based on this, the improved SuperPoint model extracts keypoint sets and descriptor vectors. Through geometric alignment, probability difference calculation, and Euclidean distance measurement of positive and negative sample pairs, it achieves learnable optimization of the keypoint detection probability threshold and non-maximum suppression radius. Furthermore, it introduces optical flow estimation, cross-frame feature mapping, and difference calculation modules, using keypoint probability differences and descriptor differences as temporal consistency constraints, effectively improving the robustness and matching accuracy of feature extraction. This method can maintain high robustness in key point detection and descriptor extraction under different printing batches and complex lighting conditions, providing high-precision and stable feature data support for subsequent page registration and positioning.

[0042] In this embodiment, S2 specifically includes: S21. Scale the target textbook page image to a uniform resolution and convert it to a grayscale image; S22. Input the grayscale image into the improved SuperPoint model, and extract the key point set and corresponding descriptor vector according to the feature extraction process; S23. The optimized learnable parameters are used to filter the key point probability heatmap to obtain a set of key points and their coordinates in the image coordinate system, and descriptor vectors are extracted at the corresponding positions. The screening process involves two steps: Iterate through each pixel position in the keypoint probability heatmap and retain only candidate positions whose keypoint detection probability is greater than or equal to the keypoint detection probability threshold. Using the non-maximum suppression radius as the neighborhood range, only the candidate positions with the highest key point detection probability are retained in each neighborhood, and other candidate positions are eliminated to obtain the final set of key points and the coordinates of the key point set in the image coordinate system. Descriptor vectors are extracted at the corresponding positions. S24. Determine the reference point of the page logical coordinate system of the target textbook page, and calculate the homography geometric transformation matrix from the image coordinate system to the page logical coordinate system. S25. Based on the homography geometric transformation matrix, transform the image coordinates of the key point set into page logical coordinates to obtain the feature point set in the page logical coordinate system; S26. Assign an index identifier to each feature point in the feature point set and establish a positional correspondence with the page content index to generate a layout template. The layout template includes page logical coordinate system parameters, a feature point set, an index identifier for each feature point, and its descriptor vector.

[0043] This invention preprocesses the target textbook page image into a uniform resolution grayscale image and inputs it into an improved SuperPoint model to extract a set of key points and corresponding descriptor vectors. Optimized learnable parameters, combined with a key point detection probability threshold and a non-maximum suppression radius, are used for high-precision screening to ensure the retention of the most discriminative feature points. Based on this, the homography geometric transformation matrix from the image coordinate system to the page logical coordinate system is calculated using a reference point in the page logical coordinate system, achieving precise mapping of feature point coordinates. An index identifier is assigned to each feature point, establishing a one-to-one correspondence with the page content index, generating a layout template containing complete geometric and feature information. This provides a highly robust and consistent benchmark data structure for subsequent page matching and precise positioning.

[0044] In this embodiment, S3 specifically includes: S31. Load the camera intrinsic parameter matrix and distortion coefficient of the built-in camera of the reading pen, and set the output resolution and timestamp recording parameters; S32. Acquire the current page image frame and record the timestamp to obtain the original page image; S33. Perform distortion correction processing on the original page image based on the parameters of Zhang Zhengyou calibration method, and perform brightness normalization by adaptive histogram equalization to obtain the corrected page image. S34. Input the corrected page image into the improved SuperPoint model, extract the key point set and corresponding descriptor vector according to the feature extraction process, and update the timing state in combination with the timing processing unit. S35. Use the optimized keypoint detection probability threshold and non-maximum suppression radius to filter the keypoint set of the current frame, perform keypoint probability heatmap filtering operation on the current frame, and obtain the two-dimensional coordinates of the keypoint set in the image coordinate system and the corresponding confidence level.

[0045] S36. Extract descriptor vectors from the key point set locations, and normalize each descriptor vector according to its vector magnitude to obtain normalized descriptors. Output the two-dimensional coordinates of the key points in the current frame in the image coordinate system, the corresponding confidence score, and the normalized descriptor.

[0046] This invention loads the camera intrinsic parameter matrix and distortion coefficients of the built-in camera of the reading pen and sets the resolution and timestamp parameters. It then acquires the current page image frame and performs distortion correction and brightness normalization processing to ensure the geometric accuracy and illumination balance of the input image. Subsequently, the corrected page image is input into an improved SuperPoint model, which combines a temporal processing unit to extract the key point set and corresponding descriptor vectors. Optimized key point detection probability thresholds and non-maximum suppression radii are used for high-precision filtering to obtain the two-dimensional coordinates and confidence scores of the key points in the image coordinate system. Based on this, the descriptor vectors are extracted and normalized, and finally, the current frame feature data containing two-dimensional coordinates, confidence scores, and normalized descriptors is output. This provides highly stable and recognizable feature input for subsequent visual registration and accurate positioning.

[0047] In this embodiment, S4 specifically includes: S41. Read the layout template and the two-dimensional coordinates and normalized descriptors of the key points in the image coordinate system; S42. Perform nearest neighbor retrieval and matching pair filtering within the descriptor metric space to obtain a candidate matching pair set; The descriptor metric space refers to a high-dimensional vector space composed of all normalized descriptors, where the similarity between any two descriptor vectors can be calculated based on a preset Euclidean distance, which is used to measure the degree of matching between descriptors.

[0048] S43. Establish the correspondence between the current frame image coordinates and the page logical coordinates based on the candidate matching pair set; S44. Based on the correspondence, the random sampling consistency algorithm is used to estimate the online homography geometric transformation matrix from the image coordinate system to the page logical coordinate system, and the parameters are refined according to the interior point set. The inner point set is the set of matching point pairs whose error is determined to be less than or equal to a preset threshold during the online estimation of homography geometric transformation matrix. S45. Record the refined online estimated homography geometric transformation matrix as visual registration parameters and synchronize it to the temporal processing unit to update the visual registration prior parameters.

[0049] This invention rapidly obtains a set of high-confidence candidate matching pairs by reading the two-dimensional coordinates and normalized descriptors of the page template and key points, performing nearest neighbor retrieval and matching pair filtering within the descriptor metric space. Based on the candidate matching pairs, it establishes the correspondence between the current frame image coordinates and the page logical coordinates, and uses a random sampling consensus algorithm to estimate the homography geometric transformation matrix online. Based on the preliminary interior point set, it uses least squares optimization to minimize the reprojection error, thereby refining the parameters. Finally, the refined homography geometric transformation matrix is ​​recorded as visual registration parameters and synchronized to the temporal processing unit, thus significantly improving registration accuracy and robustness, and providing highly reliable geometric constraints for subsequent positioning and fusion.

[0050] In this embodiment, the specific steps of parameter refinement include: Based on the correspondence, a random sampling consensus algorithm is used to estimate the online homography geometric transformation matrix from the image coordinate system to the page logical coordinate system. In the initially estimated set of interior points, the parameters of the homography geometric transformation matrix are refined using a least-squares optimization method, that is, minimizing the reprojection error of the interior points in the page logical coordinate system. The reprojection error is the Euclidean distance between the interior point in the image coordinate system after the matrix transformation and its corresponding point in the page logical coordinate system.

[0051] In this embodiment, S5 specifically includes: S51. Multi-channel electromagnetic signals are collected by an electromagnetic induction grid and corresponding timestamps are generated. The electromagnetic induction grid is a two-dimensional induction coil array structure deployed inside the interactive medium, used to sense the high-frequency electromagnetic signals emitted by the pen tip when it approaches or is touched; the signal is converted into a multi-channel electromagnetic signal by the built-in acquisition module and timestamped for subsequent electromagnetic positioning calculations.

[0052] S52. Preprocess the multi-channel electromagnetic signal and calculate the original electromagnetic coordinates in the device coordinate system; The preprocessing includes: performing bandpass filtering on the sensing signals of each channel to suppress environmental electromagnetic interference and high-frequency noise; performing synchronization alignment based on the sampling clock to ensure the consistency of data time for each channel; normalizing the amplitude of the synchronized signal to eliminate the sensitivity differences between the sensing coils; and, after the preprocessing is completed, using a coordinate inversion algorithm to calculate the original electromagnetic coordinates of the reading pen in the device coordinate system based on the amplitude distribution and phase difference of the sensing signals of each channel.

[0053] S53. During the session initialization phase, calculate the initial electromagnetic prior registration parameters from the device coordinate system to the page logical coordinate system based on the known page logical coordinate system reference point. S54. Apply the online estimated homography geometric transformation matrix to the coordinates of the key point set obtained by the improved SuperPoint model in the current frame, and calculate the position of the key points in the page logical coordinate system; The positions of key points after visual registration and transformation are compared with the positions of the original electromagnetic coordinates under the corresponding timestamp in the page logical coordinate system, and the coordinate difference vector in the page logical coordinate system is calculated. Based on the coordinate difference vector, the registration parameter update operation is performed to adjust the translation, rotation and scale components in the initial electromagnetic prior registration parameters; S55. After the update is completed, the adjusted electromagnetic prior registration parameters are used as the valid registration parameters for the current running cycle and recorded in the registration parameter cache for real-time mapping of electromagnetic coordinates.

[0054] This invention collects high-frequency electromagnetic signals emitted by the pen tip through an electromagnetic induction grid and generates multi-channel electromagnetic signals with timestamps. After preprocessing such as bandpass filtering, synchronization alignment, and amplitude normalization, the original electromagnetic coordinates in the device coordinate system are calculated using a coordinate inversion algorithm. During the session initialization phase, initial electromagnetic prior registration parameters are established using known page logical coordinate system reference points. The online estimated homography geometric transformation matrix is ​​applied to the key point set extracted by the improved SuperPoint model to calculate the position of the key points in the page logical coordinate system. The difference analysis is performed with the original electromagnetic coordinates at the same timestamp, and the translation, rotation, and scale components are dynamically updated based on the difference vector to generate and cache the effective electromagnetic registration parameters for the current operating cycle. This achieves accurate fusion mapping of visual and electromagnetic information, significantly improving the real-time performance and spatial accuracy of the pen positioning.

[0055] In this embodiment, the page logical coordinate system reference point refers to a set of calibration points that are predefined and fixed in position in the page logical coordinate system, used to establish the geometric mapping relationship between the page logical coordinate system and the device coordinate system and the image coordinate system.

[0056] The reference point of the page logical coordinate system can be selected from the set of feature points in the page layout template. It is located at the four corners, edges or other stable printing mark positions that do not change with the page batch. Its two-dimensional coordinates in the page logical coordinate system are known constants and can be used as target coordinates in the registration calculation.

[0057] In this embodiment, S6 specifically includes: S61. Read the visual registration parameters and the statistics of the matching inlier rate and reprojection error of the visual registration parameters; read the electromagnetic original coordinates, timestamp, and electromagnetic prior registration parameters. S62. Statistically analyze the original electromagnetic coordinates within the sliding time window and estimate the electromagnetic noise variance using the sample variance calculation method. S63. Calculate the set of weight coefficients based on the visual matching inlier rate, reprojection error variance and electromagnetic noise variance. S64. Solve for the fusion registration parameters by combining the visual registration parameters and the electromagnetic prior registration parameters according to the set of weight coefficients. S65. Convert the original electromagnetic coordinates into page logical coordinates using the fusion registration parameters to generate a positioning coordinate sequence corresponding to the timestamp; S66. Output the final positioning coordinate sequence and fusion registration parameters, and synchronize the fusion registration parameters to the timing processing unit as visual registration prior parameters.

[0058] This invention reads visual registration parameters and their matching inlier rate, reprojection error statistics, as well as electromagnetic original coordinates, timestamps, and electromagnetic prior registration parameters. Within a sliding time window, it estimates the electromagnetic noise variance using the sample variance method. The visual matching inlier rate, reprojection error variance, and electromagnetic noise variance are then normalized and their reciprocals normalized, respectively, to calculate a set of weighted coefficients. Based on this set of weighted coefficients, the visual registration parameters and electromagnetic prior registration parameters are weighted and fused to generate fused registration parameters. The electromagnetic original coordinates are converted into page logical coordinates, forming a positioning coordinate sequence corresponding to the timestamp. Finally, the fused registration parameters are output and synchronized to the timing processing unit as new visual registration prior parameters. This achieves adaptive weighted fusion of multi-source information, improving the stability and accuracy of positioning coordinates in dynamic environments.

[0059] In this embodiment, the set of weighting coefficients includes: The visual matching in-point rate is normalized proportionally to obtain the first weighting factor. After taking the reciprocals of the reprojection error variance and the electromagnetic noise variance, proportional normalization is performed to obtain the second weighting factor and the third weighting factor, respectively. The first weight factor, the second weight factor, and the third weight factor are combined according to the proportional coefficient to obtain a set of weight coefficients, which are used for the weighted fusion of registration parameters.

[0060] In this embodiment, the improved SuperPoint model operates according to the following specific steps. It receives single-channel grayscale page images or continuous page video frames with uniform resolution; normalizes the numerical range of the input; and organizes the input into batches according to the tensor format set by the model.

[0061] The input is subjected to a cascade of multi-layer convolution, batch normalization and non-linear activation; downsampling is performed in several layers using stride or pooling to obtain multi-scale layout structure representation; and a two-dimensional feature map containing spatial location information and local texture information is output at the end of the encoder for use by subsequent branches and temporal processing units.

[0062] When the input is a series of consecutive video frames, the pixel-level optical flow field between adjacent frames is calculated; the key point probability heatmap of the previous frame and the descriptor vector set are spatially mapped according to the optical flow field to generate cross-frame alignment results.

[0063] Based on the cross-frame alignment results and the current frame's two-dimensional feature map, a keypoint probability difference map and a descriptor difference matrix are generated. During the training phase, the difference results are sent to the training control module for temporal consistency constraints. During the inference phase, they are only used to update the temporal state and do not participate in loss calculation.

[0064] It receives a 2D feature map, performs convolution and probability mapping operations, and outputs a keypoint probability heatmap. The heatmap establishes a fixed-scale mapping relationship with the input image in the spatial dimension, supporting subsequent coordinate conversion.

[0065] First, thresholding based on the keypoint detection probability threshold is performed on the keypoint probability heatmap to obtain a set of candidate locations. Then, local extrema are filtered in the neighborhood of each candidate location according to the non-maximum suppression radius, and the keypoint set, image coordinates and confidence score are output. The keypoint detection probability threshold and the non-maximum suppression radius are updated together with the network weights as learnable parameters during training, and participate in the filtering as fixed parameters during inference.

[0066] The system receives a two-dimensional feature map, performs convolution, batch normalization, and non-linear activation operations to generate a set of descriptor vectors that correspond one-to-one with the spatial locations of the two-dimensional feature map; it then extracts descriptor vectors from the keypoint set locations and normalizes them according to the vector magnitude to obtain normalized descriptors.

[0067] When alignment with the input resolution is required, the keypoint probability heatmap and descriptor grid are upsampled and reconstructed by coordinate mapping to ensure that the keypoint coordinates are consistent with the page image coordinate system, which facilitates subsequent mapping with the layout template or page logical coordinate system.

[0068] During the training phase, the system receives stable feature point labels from manually annotated page samples, supervision signals generated by homography geometric alignment, descriptor positive and negative sample pair information, and temporal difference results generated in step four. Based on keypoint classification supervision, descriptor matching metrics, and temporal consistency constraints, the system updates the trainable parameters of the shared feature encoder, keypoint detection branch, and descriptor extraction branch, and simultaneously updates the two learnable parameters: keypoint detection probability threshold and non-maximum suppression radius. During the inference phase, no parameter updates are performed; only the keypoint set, confidence score, and normalized descriptor are output according to steps five through seven.

[0069] The improved SuperPoint model of this invention normalizes and batches grayscale page images or continuous video frames of uniform resolution, extracts multi-scale page structure features by combining multi-layer convolution, batch normalization, and nonlinear activation operations, and utilizes optical flow fields to achieve cross-frame feature mapping in continuous frame scenarios to construct temporal consistency constraints. In the keypoint detection branch, the model achieves high-precision keypoint selection through learnable probability thresholds and non-maximum suppression radii. In the descriptor extraction branch, it generates normalized descriptors to ensure the stability and comparability of matching, while supporting coordinate mapping and resolution alignment between keypoints and descriptors. During the training phase, it jointly optimizes the network parameters by combining keypoint classification supervision, descriptor matching metrics, and temporal consistency constraints. During the inference phase, it efficiently outputs keypoint sets, confidence scores, and normalized descriptors, thus balancing the accuracy of feature extraction with the stability of cross-frame tracking, providing a reliable feature foundation for subsequent page registration and electromagnetic coordinate fusion.

[0070] Example 1: To verify the feasibility of this invention in practice, it was applied to an interactive system between a smart reading pen and accompanying teaching materials from an educational equipment company. This system is designed for various types of printed pages, including lower elementary school literacy textbooks, middle school physics and chemistry experiment guides, and high school comprehensive review materials. It requires accurate page position recognition and rapid association of text and image content even when the user moves the reading pen quickly. In existing technologies, pure electromagnetic positioning is prone to significant errors at page edges and on highly reflective materials, while pure visual positioning lacks stability under complex lighting conditions, especially exhibiting noticeable positioning drift after prolonged use, leading to decreased reading accuracy and a poor user experience. This invention, by integrating improved SuperPoint visual feature extraction with multi-channel electromagnetic positioning data, combined with registration parameter optimization and temporal consistency processing, significantly improves positioning accuracy and stability under different materials, lighting conditions, and batch printing conditions.

[0071] In application, the system first uses the built-in camera of the reading pen to capture page image frames, extracts key point sets and descriptors through an improved SuperPoint model, and generates a layout template in the page's logical coordinate system. Simultaneously, the electromagnetic induction module at the bottom of the reading pen collects multi-channel electromagnetic signals in real time and calculates them into electromagnetic primitive coordinates in the device coordinate system. Then, through weighted fusion of visual registration parameters and electromagnetic prior registration parameters, the electromagnetic coordinates are mapped to the page's logical coordinate system, and combined with timestamps to generate the final positioning coordinate sequence, achieving precise matching between the page content and the pen tip position. In testing, to simulate actual usage environments, three types of page materials were selected: ordinary offset paper, laminated cardboard, and high-reflectivity coated paper. Positioning accuracy and stability were tested under different lighting conditions, including natural light, strong direct light, and low indoor light. Furthermore, to evaluate cross-batch adaptability, batch switching tests were conducted on the same textbook version printed in multiple batches, and the positioning drift was recorded after 5 hours of continuous use to examine long-term operational stability. Specific data are shown in Table 1. Table 1. Comparison of positioning accuracy and stability between the present invention and traditional methods under different test conditions.

[0072] After three months of systematic testing, the positioning accuracy of this invention demonstrated a significant advantage across various page types, including lower elementary literacy textbooks, middle school physics and chemistry experiment guides, and high school comprehensive review materials. In traditional, non-fusion electromagnetic positioning schemes, the average positioning error in the central area of ​​the page was approximately 1.8 mm, while the average positioning error in the edge area reached 3.5 mm. However, using the method of this invention, the average positioning error in the central area was reduced to 0.9 mm, and in the edge area to 1.5 mm, resulting in an overall accuracy improvement of over 50%. In tests on highly reflective laminated cardboard pages, the average error of traditional purely visual methods was 2.7 mm, while the fusion scheme of this invention reduced the error to 1.2 mm, improving stability by over 55%. Furthermore, in long-term usage stability tests, the positioning drift after 5 hours of continuous use decreased from 2.3 mm in the original scheme to 0.8 mm, proving that this invention maintains high-precision positioning capabilities even under long-term usage conditions. Under low-reflectivity ordinary offset paper and natural lighting conditions, the center area error of the traditional electromagnetic positioning scheme is 1.8 mm and the edge area error is 3.4 mm, while the center area error of the method of this invention is 0.9 mm and the edge area error is 1.5 mm. Under high-reflectivity laminated cardboard and strong lighting conditions, the center area error of the traditional visual scheme is 2.7 mm and the edge area error is 4.1 mm, while the center area error of the method of this invention is 1.2 mm and the edge area error is 2.0 mm. In pages with differences in printing from multiple batches, the traditional method has an average error fluctuation of ±1.5 mm when switching batches, while the method of this invention controls the error within ±0.5 mm.

[0073] The data in Table 1 fully demonstrate that the present invention not only has a significant improvement in accuracy, but also has higher stability and adaptability under conditions of cross-batch use, different materials, and long-term use.

[0074] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for locating an electromagnetic reading pen based on image recognition, characterized in that, Includes the following steps: S1. Collect images of the textbook pages and perform preprocessing sequentially. Combine the improved SuperPoint model and time-series processing unit to complete feature extraction, alignment, difference calculation and learnable parameter optimization of the key point set and descriptor vector. S2. Extract the set of key points from the target textbook page image, calculate the homography geometric transformation matrix by combining the reference points of the page logical coordinate system, complete the coordinate transformation and generate the page template; S3. Collect and correct the current page image frame, extract and filter the key point set, and generate current frame feature data containing two-dimensional coordinates, confidence and normalized descriptors; S4. Based on the matching of the layout template and the feature data of the current frame, establish a correspondence, use the random sampling consistency algorithm to estimate and refine the homography geometric transformation matrix, and generate visual registration parameters; S5. Combine the visual registration results with the comparison and update the initial electromagnetic prior registration parameters, and output the effective registration parameters for the current running cycle. S6. Generate the positioning coordinate sequence and update the visual registration prior parameters.

2. The electromagnetic reading pen positioning method based on image recognition according to claim 1, characterized in that, The feature extraction process includes: The input image is fed into a shared encoder and generated as an initial two-dimensional feature map through the first convolutional layer. Batch normalization is applied at the output of each convolutional layer, and a non-linear activation function is applied. During the stacking of convolutional layers, the spatial resolution of the initial two-dimensional feature map is gradually reduced and the channel dimension is increased through downsampling operations, thereby encoding multi-scale layout structure features; The shared encoder outputs a two-dimensional feature map containing spatial location information and local texture information at its end. The two-dimensional feature map output by the shared encoder is input into the key point detection branch and the descriptor extraction branch respectively to generate a key point probability heatmap and a set of descriptor vectors at the corresponding positions. The keypoint probability heatmap is filtered, and the set of keypoints and their coordinates in the image coordinate system are output. Descriptor vectors are extracted at the corresponding positions.

3. The electromagnetic reading pen positioning method based on image recognition according to claim 1, characterized in that, S1 specifically includes: S11. Collect images of textbook pages and perform preprocessing, including: unified resolution conversion, geometric transformation and physical enhancement processing, to generate enhanced image pairs; S12. Input the enhanced image to the improved SuperPoint model and extract the key point set and corresponding descriptor vectors according to the feature extraction process; S13. For one page image in the enhanced image pair, the key point probability heatmap of the other page image is geometrically aligned according to the homography geometric transformation matrix, and the probability difference is calculated at the alignment position. S14. In the enhanced image pair, extract the descriptor vectors for the corresponding positions after geometric alignment, calculate the Euclidean distance between the positive sample pair and the negative sample pair, minimize the distance of the positive sample pair and maximize the distance of the negative sample pair. S15. In the manually annotated textbook page samples, the classification training of the keypoint probability heatmap is guided by the real keypoint labels. The keypoint detection probability threshold and the non-maximum suppression radius are set as learnable parameters and iteratively updated along with the weights of the improved SuperPoint model until convergence, thus obtaining the optimized learnable parameters. Among them, the keypoint detection probability is generated by the keypoint detection branch of the improved SuperPoint model, which represents the probability value of each pixel position belonging to the keypoint category. S16. A temporal processing unit is introduced between the shared encoder and the key point detection branch and the descriptor extraction branch. The temporal processing unit includes an optical flow estimation module, a cross-frame feature mapping module and a difference calculation module. The continuously acquired video frames of the textbook pages are sequentially input into the shared encoder of the improved SuperPoint model to obtain two-dimensional feature maps of adjacent frames; The pixel-level optical flow field between adjacent frames is calculated based on the optical flow estimation module. The key point probability heatmap and descriptor vector set of the previous frame are mapped to the corresponding spatial location of the next frame through the cross-frame feature mapping module. In the difference calculation module, based on the corresponding spatial location, the difference between the keypoint detection probability after mapping and the keypoint detection probability at the corresponding location in the next frame is calculated position by position to obtain the keypoint probability difference map; the Euclidean distance between the mapped descriptor vector and the descriptor vector at the corresponding location in the next frame is calculated to obtain the descriptor difference matrix.

4. The electromagnetic reading pen positioning method based on image recognition according to claim 1, characterized in that, S2 specifically includes: S21. Scale the target textbook page image to a uniform resolution and convert it to a grayscale image; S22. Input the grayscale image into the improved SuperPoint model, and extract the key point set and corresponding descriptor vector according to the feature extraction process; S23. The optimized learnable parameters are used to filter the key point probability heatmap to obtain a set of key points and their coordinates in the image coordinate system, and descriptor vectors are extracted at the corresponding positions. The screening process involves two steps: Iterate through each pixel position in the keypoint probability heatmap and retain only candidate positions whose keypoint detection probability is greater than or equal to the keypoint detection probability threshold. Using the non-maximum suppression radius as the neighborhood range, only the candidate positions with the highest key point detection probability are retained in each neighborhood, and other candidate positions are eliminated to obtain the final set of key points and the coordinates of the key point set in the image coordinate system. Descriptor vectors are extracted at the corresponding positions. S24. Determine the reference point of the page logical coordinate system of the target textbook page, and calculate the homography geometric transformation matrix from the image coordinate system to the page logical coordinate system. S25. Based on the homography geometric transformation matrix, transform the image coordinates of the key point set into page logical coordinates to obtain the feature point set in the page logical coordinate system; S26. Assign an index identifier to each feature point in the feature point set and establish a positional correspondence with the page content index to generate a layout template. The layout template includes page logical coordinate system parameters, a feature point set, an index identifier for each feature point, and a descriptor vector.

5. The electromagnetic reading pen positioning method based on image recognition according to claim 1, characterized in that, S3 specifically includes: S31. Load the camera intrinsic parameter matrix and distortion coefficient of the built-in camera of the reading pen, and set the output resolution and timestamp recording parameters; S32. Acquire the current page image frame and record the timestamp to obtain the original page image; S33. Perform distortion correction processing on the original page image based on the parameters of Zhang Zhengyou calibration method, and perform brightness normalization by adaptive histogram equalization to obtain the corrected page image. S34. Input the corrected page image into the improved SuperPoint model, extract the key point set and corresponding descriptor vector according to the feature extraction process, and update the timing state in combination with the timing processing unit. S35. Use the optimized keypoint detection probability threshold and non-maximum suppression radius to filter the keypoint set of the current frame, perform keypoint probability heatmap filtering operation on the current frame, and obtain the two-dimensional coordinates of the keypoint set in the image coordinate system and the corresponding confidence level. S36. Extract descriptor vectors from the keypoint set locations, and normalize each descriptor vector according to its vector magnitude to obtain a normalized descriptor; output the two-dimensional coordinates of the keypoints in the current frame in the image coordinate system, the corresponding confidence scores, and the normalized descriptors to form the feature data of the current frame.

6. The electromagnetic reading pen positioning method based on image recognition according to claim 1, characterized in that, S4 specifically includes: S41. Read the layout template and the two-dimensional coordinates and normalized descriptors of the key points in the image coordinate system; S42. Before performing nearest neighbor retrieval and matching pair filtering in the descriptor metric space, based on the descriptor difference matrix calculated at the corresponding spatial location by the time series processing unit, key points whose descriptor differences exceed the set threshold are removed, and the set of key points that satisfy the difference constraints is retained; nearest neighbor retrieval and matching pair filtering are performed in the set of key points to obtain a set of candidate matching pairs. The descriptor metric space refers to a high-dimensional vector space composed of all normalized descriptors, where the similarity between any two descriptor vectors can be calculated based on a preset Euclidean distance, which is used to measure the degree of matching between descriptor vectors. S43. Establish the correspondence between the current frame image coordinates and the page logical coordinates based on the candidate matching pair set; S44. Based on the correspondence, the random sampling consistency algorithm is used to estimate the online homography geometric transformation matrix from the image coordinate system to the page logical coordinate system, and the parameters are refined according to the interior point set. The inner point set is the set of matching point pairs whose error is determined to be less than or equal to a preset threshold during the online estimation of homography geometric transformation matrix. S45. Record the refined online estimated homography geometric transformation matrix as visual registration parameters and synchronize it to the temporal processing unit to update the visual registration prior parameters. Based on the key point probability difference map output by the temporal processing unit, calculate the proportion of high difference regions. If it exceeds the set difference threshold, apply confidence weight adjustment to the current visual registration parameters and use the adjustment result as a weight factor to synchronously adjust the update amplitude of the electromagnetic prior registration parameters.

7. The electromagnetic reading pen positioning method based on image recognition according to claim 1, characterized in that, S5 specifically includes: S51. Multi-channel electromagnetic signals are collected by an electromagnetic induction grid and corresponding timestamps are generated. S52. Preprocess the multi-channel electromagnetic signal and calculate the original electromagnetic coordinates in the device coordinate system; S53. During the session initialization phase, calculate the initial electromagnetic prior registration parameters from the device coordinate system to the page logical coordinate system based on the known page logical coordinate system reference point. S54. Apply the online estimated homography geometric transformation matrix to the coordinates of the key point set obtained by the improved SuperPoint model in the current frame, and calculate the position of the key points in the page logical coordinate system; The positions of key points after visual registration and transformation are compared with the positions of the original electromagnetic coordinates under the corresponding timestamp in the page logical coordinate system, and the coordinate difference vector in the page logical coordinate system is calculated. Based on the coordinate difference vector, the registration parameter update operation is performed to adjust the translation, rotation and scale components in the initial electromagnetic prior registration parameters; S55. After the update is completed, the adjusted electromagnetic prior registration parameters are used as the valid registration parameters for the current running cycle and recorded in the registration parameter cache for real-time mapping of electromagnetic coordinates.

8. The electromagnetic reading pen positioning method based on image recognition according to claim 1, characterized in that, S6 specifically includes: S61. Read the visual registration parameters and the statistics of the matching inlier rate and reprojection error of the visual registration parameters; read the electromagnetic original coordinates, timestamp, and electromagnetic prior registration parameters. S62. Statistically analyze the original electromagnetic coordinates within the sliding time window and estimate the electromagnetic noise variance using the sample variance calculation method. S63. Calculate the set of weight coefficients based on the visual matching inlier rate, reprojection error variance and electromagnetic noise variance. S64. Solve for the fusion registration parameters by combining the visual registration parameters and the electromagnetic prior registration parameters according to the set of weight coefficients. S65. Convert the original electromagnetic coordinates into page logical coordinates using the fusion registration parameters to generate a positioning coordinate sequence corresponding to the timestamp; S66. Output the final positioning coordinate sequence and fusion registration parameters, and synchronize the fusion registration parameters to the timing processing unit as visual registration prior parameters.

Images