Learning method based on multiple sensors, real-time visual SLAM and AR

Through multi-sensor data fusion and deep learning algorithms, the problem of data acquisition inconsistency between visual SLAM and LBA systems is solved, high-precision pose map optimization and AR content generation are realized, and the authenticity and interactivity of the learning experience are improved.

CN119068152APending Publication Date: 2024-12-03NORTHWEST NORMAL UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411301693.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-18
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In the prior art, the inconsistency between visual SLAM and LBA systems in data acquisition frequency and accuracy makes it difficult to achieve high-precision positioning and map construction in a dynamic environment, affecting the accurate superposition and learning experience of AR content.

Method used

Using multi-sensor data fusion, synchronize sensor visual data and GPS data through timestamp interpolation and Kalman filter, combine machine learning and deep learning algorithms for feature extraction, matching and AR content generation, and use sparse optimization and semantic segmentation algorithm to achieve high precision pose map optimization and environment perception.

Benefits of technology

It improves positioning accuracy and robustness, ensures consistency of data fusion, improves image feature extraction speed and matching accuracy, realizes high-precision perception and dynamic update of virtual content, and enhances the authenticity and interactivity of learning experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure SMS_1
    Figure SMS_1
  • Figure SMS_5
    Figure SMS_5
  • Figure SMS_8
    Figure SMS_8
Patent Text Reader

Abstract

The invention provides a learning method based on multiple sensors, real-time visual SLAM (Simultaneous Localization and Mapping) and AR (Augmented Reality), which is characterized in that visual SLAM and GPS (Global Positioning System) data are fused through a timestamp interpolation method and an extended Kalman filter, so that the positioning precision and robustness are improved, and abnormities in sensor data are detected and corrected in real time by utilizing a machine learning algorithm; a deep learning assisted feature extraction and matching algorithm is used to improve the image feature extraction speed and matching precision, and a sparse optimization method is used to optimize the pose image to improve the precision and robustness of the system; the deep learning algorithm is used for object recognition and tracking, the recognition and tracking precision is improved, the recognition and tracking speed is increased, the high-precision semantic segmentation algorithm is adopted, high-precision perception and semantic understanding of the environment are achieved, the dynamic updating algorithm is adopted, virtual content can be updated in real time according to environment changes, and the authenticity and interactivity of learning experience are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence and relates to a learning device based on multi-sensors, real-time visual SLAM optimization and AR content. Background Art

[0002] When learning a foreign language, learners need to be placed in the target language environment and interact with it in various ways. Location-based augmented reality (LBA) can link foreign language learning resources with real-life situations, while visual SLAM can achieve real-time positioning and map construction of physical space. AR technology can then be used to achieve a seamless integration of digital resources and physical space.

[0003] Currently, visual SLAM relies on visual sensors, while LBA systems usually rely on positioning systems such as GPS, sensor data synchronization and calibration, and the different acquisition frequencies and accuracies of visual data and GPS data. It is necessary to develop synchronization mechanisms and calibration algorithms to ensure consistency during data fusion. In addition, visual SLAM needs to build maps and locate in real time in dynamic environments, which places high demands on the real-time and computational efficiency of the algorithm. AR content generation and positioning, augmented reality technology needs to accurately superimpose virtual learning resources into the physical environment, which requires high-precision positioning and dynamic environment perception capabilities. Summary of the invention

[0004] The purpose of the present invention is to provide a learning device based on multi-sensor, real-time visual SLAM optimization and AR content to address the problems existing in the prior art.

[0005] To this end, the present invention adopts the following technical solutions: A learning method based on multi-sensor, real-time visual SLAM and AR, comprising the following steps: (1) Multi-sensor data fusion: using a timestamp-based interpolation method to synchronize sensor visual data and GPS data, and then using a Kalman filter to synchronize and fuse sensor visual data and GPS data; (2) Real-time visual SLAM optimization, using convolutional neural networks for feature extraction and matching, extracting feature points and matching adjacent feature points using the Euclidean distance or Hamming distance of the descriptor, and using sparse optimization methods to optimize the pose graph and minimize the error function; (3) AR content generation: using deep learning-based object recognition and tracking algorithms to quickly track and extract objects in images, and then using semantic segmentation algorithms to perform high-precision perception and semantic understanding of the virtual environment. Finally, the virtual content can be updated in real time according to environmental changes.

[0006] Furthermore, in the step (1), a machine learning algorithm is used to detect and process abnormal sensor vision data when the sensor vision data and the GPS data are synchronously integrated.

[0007] Furthermore, the machine learning algorithms are support vector machine and random forest.

[0008] Furthermore, in step (2), a priority-based computing resource allocation strategy is used to perform feature extraction and matching.

[0009] Furthermore, the object recognition and tracking algorithm based on deep learning in step (3) is YOLO or Mask R-CNN.

[0010] The beneficial effects of the present invention are: (1) The visual SLAM data is synchronized and calibrated with the GPS data through the timestamp interpolation method to ensure the consistency of data fusion. The extended Kalman filter (EKF) is applied to fuse the visual SLAM and GPS data to improve the positioning accuracy and robustness. The machine learning algorithm is used to detect and correct anomalies in the sensor data in real time to ensure the stable operation of the system.

[0011] (2) Use deep learning-assisted feature extraction and matching algorithms to improve the speed of image feature extraction and matching accuracy. Use sparse optimization methods to optimize the pose graph to improve the accuracy and robustness of the system. Use a priority-based computing resource allocation strategy to ensure the real-time performance of the SLAM algorithm under limited computing resources.

[0012] (3) Use deep learning algorithms for object recognition and tracking to improve the accuracy and speed of recognition and tracking; adopt high-precision semantic segmentation algorithms to achieve high-precision perception and semantic understanding of the environment; and adopt dynamic update algorithms to enable virtual content to be updated in real time according to environmental changes, thereby improving the authenticity and interactivity of the learning experience. DETAILED DESCRIPTION

[0013] The present invention is described in detail below in conjunction with embodiments: A learning method based on multi-sensor, real-time visual SLAM and AR, comprising the following steps: (1) Multi-sensor data fusion: In this example, sensor visual data collected from 69 students during their learning process is used. V t and GPS data G t , the sensor visual data and GPS data are synchronized using a timestamp-based interpolation method. The algorithm model for data synchronization is: in, is the interpolated GPS data, t is the timestamp of the sensor visual data, The timestamp is t i and t i+1 GPS data and t i ≦ t ≦ t i+1 , through the timestamp-based interpolation method, the sensor vision data can be V t Interpolation at the time point corresponding to the timestamp , in order to achieve data synchronization, through the interpolation method, solve the inconsistency problem of sensor visual data and GPS data in acquisition frequency and accuracy, align the data in time, synchronize and calibrate the sensor visual data with the GPS data, ensure the consistency of data fusion, and then use the Kalman filter to synchronize the sensor visual data and GPS data. The algorithm model is as follows: X k = F k-1 X k-1 + B k-1 U k-1 + W k-1 Z k = H k X k + V k in, X k For the states containing position and velocity, Z k is the observation vector containing sensor vision data and GPS data, F k-1 is the state transfer matrix, B k-1 is the control input matrix, U k-1 is the control input, W k-1 is process noise, which is used to describe the random disturbance in the system process. H k is the observation matrix, which is used to describe the mapping relationship from system state to observation value. V kis the observation noise, which is used to describe the random disturbance in the observation process.

[0014] The prediction steps of the algorithm model are: in, is the predicted system state vector, is the predicted state covariance matrix, which is used to describe the uncertainty of state estimation. Q k-1 is the process noise covariance matrix.

[0015] The update steps of the algorithm model are: in, K k is the Kalman gain matrix, which determines the weights between the predicted state and the observed value, is the updated system state vector, is the updated state covariance matrix, R k is the observation noise covariance matrix, is the transpose of the observation matrix, I is the unit matrix, and the sensor visual data and GPS data are fused through the Kalman filter to provide more accurate positioning and map construction. The Kalman filter dynamically adjusts the system state and uncertainty through prediction and update steps, thereby improving the robustness and accuracy of the system.

[0016] In addition, when the sensor visual data and GPS data are synchronously integrated, a machine learning algorithm is used to detect and process the sensor visual abnormal data. In this embodiment, the machine learning algorithm is a support vector machine and a random forest. Through anomaly detection and correction, anomalies in the sensor data are identified and processed to improve the reliability of data fusion and the stability of the system. Specifically, anomaly detection is achieved by combining statistical analysis methods with machine learning algorithms. The observation residual is set. r k for: By analyzing the statistical characteristics of the residuals, abnormal data can be detected: in, is the set anomaly detection threshold. The detected anomaly data can be corrected by resampling and interpolation methods.

[0017] (2) Real-time visual SLAM optimization. First, a convolutional neural network is used for feature extraction and matching. Feature points are extracted and adjacent feature points are matched using the Euclidean distance or Hamming distance of the descriptor. Specifically, the image frame is setI t exist t At the point, feature points of the image are extracted through feature extraction algorithms such as ORB, SIFT, and SURF. f t ,and f t =FeatureExtact( I t ), the feature point descriptor is d t ,and d t =Descriptor( I t, f t ), match the feature points of adjacent frames, and use the Euclidean distance or Hamming distance of the descriptor for matching: In order to improve processing speed and matching accuracy, feature extraction and matching algorithms based on deep learning (such as SuperPoint, D2-Net, etc.) are introduced to extract feature points and descriptors through convolutional neural networks (CNN): f t ,d t =CNN( I t ) I t For time t The image frame at f t For time t The feature points extracted at d t For time t The feature point descriptor at m t,t - 1 For time t and t -1 feature point matching pairs, through the feature extraction and matching algorithm based on deep learning, improve the speed of feature extraction and matching accuracy, and enhance the real-time performance of the system.

[0018] Then, the pose graph can be optimized using a sparse optimization method to minimize the error function. In this embodiment, a sparse optimization method such as g2o or Ceres Solver can be used to optimize the pose graph. The optimization goal is to minimize the error function. Specifically, the pose graph is set ,in n Represents the fixed point of posture, ε Represents the edge, the algorithm model is as follows: in, X ={ X i} represents all poses, Z ij represents the observed value, h ( X i , X j ) represents the predicted value, represents the information matrix.

[0019] After optimization, the efficiency and accuracy of graph optimization can be improved and the robustness of the SLAM system can be enhanced.

[0020] In addition, a priority-based computing resource allocation strategy is used for feature extraction and matching, and computing resources are allocated to key tasks to ensure the real-time performance of the SLAM algorithm under limited computing resources. Specifically, the priority queue is first set , q i Representation Task i Priority, that is, first clarify all the tasks involved in the SLAM algorithm, such as front-end visual odometer, back-end optimization, loop detection, map construction, etc., and then evaluate the priority for each task. i Assign a priority q i Finally, a priority queue is constructed to sort all tasks from high to low (or low to high, depending on the specific implementation) and put them into a priority queue. This queue will be used for subsequent resource allocation decisions, and then computing resources can be allocated according to priority. R i , the allocation is as follows: in, R total Represents total computing resources, ensuring real-time performance of critical tasks by dynamically adjusting priorities and resource allocation. R i For the task i Allocated computing resources.

[0021] (3) AR content generation: Use deep learning-based object recognition and tracking algorithms. In this embodiment, YOLO or Mask R-CNN is used to quickly track and extract objects in the image. Specifically, the input image is set to I t , the target detection model is Detector , output object bounding box b t and category labels ct : b t , c t = Detector ( I t ) Object tracking uses a multi-target tracking algorithm based on the Kalman filter, setting the target state vector as X k , the observation vector is Z k , the state transfer equation and observation equation are: X k = F k-1 X k-1 ++ W k-1 Z k = H k X k + V k By combining deep learning and Kalman filters, objects can be identified and tracked in real time, improving the positioning accuracy of virtual content.

[0022] Then, the semantic segmentation algorithm is used to perform high-precision perception and semantic understanding of the virtual environment. Specifically, the input image is also set as I t , the segmentation model is Segmenter , the output is a semantic segmentation map S t: : S t = Segmenter ( I t ) Environmental perception, using semantic segmentation maps S t Understand the environment semantically and extract the location information of key objects P t : P t = ExtractPositions ( S t ) Semantic segmentation enables high-precision perception and semantic understanding of the environment, assisting in the precise positioning of virtual content.

[0023] Finally, the virtual content can be updated in real time according to environmental changes. Specifically, the virtual content is set V Location and physical environment P The corresponding relationship is V ( P ), the algorithm for updating virtual content in real time is: in, V t For time t Virtual content at P t For time t The physical environment location, f is a dynamic update function, △ t is the time interval, is the speed of the virtual content, and the updated virtual content position is: By dynamically updating algorithms, virtual content can be updated in real time according to environmental changes, improving the authenticity and interactivity of the learning experience.

Claims

1. A learning method based on multi-sensor, real-time visual SLAM and AR, characterized in that: The following steps are involved: (1) Multi-sensor data fusion: the sensor visual image data and GPS data are synchronized using a timestamp-based interpolation method, and then the sensor visual data and GPS data are synchronized and fused using a Kalman filter; (2) Real-time visual SLAM optimization, using convolutional neural networks for feature extraction and matching, extracting feature points and matching adjacent feature points using the Euclidean distance or Hamming distance of the descriptor, and using sparse optimization methods to optimize the pose graph and minimize the error function; (3) AR content generation: Use deep learning-based object recognition and tracking algorithms to quickly track and extract objects in images generated by the AR system. Then use semantic segmentation algorithms to perform high-precision perception and semantic understanding of the virtual environment. Finally, the virtual content can be updated in real time according to environmental changes.

2. A learning method based on multi-sensor, real-time visual SLAM and AR according to claim 1, characterized in that: In the step (1), a machine learning algorithm is used to detect and process abnormal sensor visual data when the sensor visual data and the GPS data are synchronously integrated.

3. A learning method based on multi-sensor, real-time visual SLAM and AR according to claim 1, characterized in that: The machine learning algorithms are support vector machine and random forest.

4. A learning method based on multi-sensor, real-time visual SLAM and AR according to claim 1, characterized in that: In the step (2), a priority-based computing resource allocation strategy is used to perform feature extraction and matching.

5. The learning method based on multi-sensor, real-time visual SLAM and AR according to claim 1, characterized in that: The object recognition and tracking algorithm based on deep learning in step (3) is YOLO or Mask R-CNN.