A dynamic synchronous positioning and mapping method and system for augmented reality

Abstract

The present invention belongs to the technical field of augmented reality, and discloses a dynamic synchronous positioning and mapping method and system for augmented reality. The present invention integrates binocular event cameras, binocular grayscale cameras and inertial sensor data to improve the real-time performance and accuracy of positioning and mapping. Through event noise elimination and block motion compensation, the blurring problem of traditional cameras under high-speed motion or complex lighting conditions is solved. A feature fusion strategy based on dynamic weight adjustment is innovatively proposed to fuse the event time surface with the grayscale image. The system effectively processes motion information at different scales through a variable scale joint optimization algorithm, and dynamically adjusts key frames and optimization windows to improve the robustness of the system in augmented reality scenes. The system has low computational load and high accuracy, and is suitable for augmented reality applications in real-time environments.

Description

Technical Field

[0001] The present invention relates to the field of augmented reality technology, and in particular to an augmented reality-oriented dynamic synchronous positioning and mapping method and system. Background Art

[0002] The synchronous positioning and mapping technology originated in the field of robotics. This technology can obtain the device posture in an unknown environment in real time and build a map of the environment. In the process of user interaction with augmented reality, it is necessary to match the three-dimensional space where the user is located with the space seen by the human eye, so real-time and accurate synchronous positioning and mapping technology is essential.

[0003] The simultaneous positioning and mapping technology for augmented reality is highly dependent on hardware devices, especially the type and performance of sensors, which determine the real-time and accuracy of the simultaneous positioning and mapping algorithm. However, the lighting conditions in augmented reality scenes vary greatly, and the objects move at a fast speed. Existing ordinary camera sensors have problems with motion blur and overexposure in the aforementioned scenes. In addition, the frame data generated by ordinary cameras puts a lot of pressure on the bandwidth, storage, and computing of the device, and cannot guarantee the real-time and accuracy of simultaneous positioning and mapping. This puts higher requirements on algorithm efficiency and performance optimization. The system needs to reduce energy consumption and latency as much as possible while ensuring real-time and accuracy to improve user experience.

[0004] Therefore, how to solve the problems of motion blur and overexposure of data collected by ordinary camera sensors in scenes with large changes in lighting conditions and fast object movement, and to provide a synchronous positioning and mapping data collection method and equipment with small real-time bandwidth and computing load, are technical problems that technical personnel in this field need to solve urgently. Summary of the invention

[0005] The purpose of the present invention is to provide a dynamic synchronous positioning and mapping method and system for augmented reality to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A dynamic synchronous positioning and mapping method for augmented reality, comprising the following steps:

[0008] Step 1: calibrate the internal and external parameters of the binocular event camera and the binocular grayscale camera, and perform image distortion correction;

[0009] Step 2: Use the binocular event camera and the binocular grayscale camera to collect data at the same time, obtain image frames and event streams, and perform time synchronization to ensure data consistency. The binocular grayscale camera saves the grayscale image.

[0010] Step 3, using inertial sensors to collect angular velocity and linear velocity;

[0011] Step 4, pre-integrate the angular velocity and acceleration data obtained by the inertial sensor to obtain the posture, velocity and position information of the head mounted device;

[0012] Step 5: Remove background and hotspot noise from the event stream;

[0013] Step 6: Use the angular velocity and linear velocity provided by the inertial sensor to perform block motion compensation and correct event stream distortion;

[0014] Step 7, convert the event stream into a time surface representation, extract edge features and generate an edge map image;

[0015] Step 8, extracting feature points from the grayscale image and the edge mapping image respectively, and matching them;

[0016] Step 9, using a dynamic weight model to fuse the feature point data of the event camera and the grayscale camera to construct a feature point set of the key frame;

[0017] Step 10, using the KLT optical flow tracking algorithm to track the motion trajectory of the feature points of the key frame; using the EPnP algorithm to estimate the initial posture of the head-mounted device to provide preliminary information for subsequent posture optimization;

[0018] Step 11, combining the pre-integrated data and the preliminary pose data, performing variable-scale joint optimization, and constructing a local map;

[0019] Step 12: Output the global pose and global map.

[0020] Furthermore, aligning the acquisition time of the binocular event camera and the binocular grayscale camera in step 2 includes the following process:

[0021] S100, initializing the binocular event camera data acquisition parameters, assuming that the timestamp of the latest grayscale image frame is t n , the timestamp of the next latest frame is t n-1 , set the event queue to record the time t=t between two adjacent frames of the binocular grayscale camera n -t n-1 All event streams within;

[0022] S101, for all event streams acquired within time t, import these event streams into an event queue. If the event queue is full, delete the earliest data in the event queue in turn to release event space.

[0023] Furthermore, the step 5 comprises the following steps:

[0024] Step 5.1, converting the data stored in the event queue into voxels in three-dimensional space, and recording the number of events in each voxel;

[0025] Step 5.2, count and analyze the timestamps and spatial coordinates of the events in the voxels, and remove noise events with unreasonable timestamps, repeated spatial coordinates, and small differences in event timestamps;

[0026] Step 5.3, cluster the events of each voxel according to the spatial position, merge the events in the same spatial voxel, and calculate the time mean and variance of the events in each voxel. Use the ratio of the time mean to the variance as the weight factor of the event, so as to perform weighted averaging on the events in the voxel, and the result is used as the event output of the voxel;

[0027] In step 5.4, by calculating the time mean and variance of each voxel event output, the event voxel output with a small ratio of the time mean to the variance is removed, thereby obtaining the event stream after removing the noise data.

[0028] Furthermore, the step 6 comprises the following steps:

[0029] Step 6.1, divide the event stream into multiple small blocks by time, with each block being 20-50 milliseconds. Use the angular velocity ω and linear velocity v provided by the inertial sensor to estimate the camera motion within each block, and calculate the transformation matrix R(t) and translation vector T(t) of the block within the block time window.

[0030] Step 6.2, correct the pixel coordinates (u, v) of each event in the block:

[0031] Formula (1)

[0032] Where x is the original event coordinate and x′ is the compensated coordinate.

[0033] Furthermore, the step 7 comprises the following steps:

[0034] Step 7.1, represent the event stream as a time surface, traverse the event stream, and find the coordinates of each pixel x=(u,v) T The most recent event within time t, where u is the coordinate of the pixel plane in the x-axis direction, v is the coordinate of the pixel plane in the y-axis direction, and the corresponding timestamp is t last , then t≥t last The time surface at time (x) is defined as:

[0035] Formula (2)

[0036] Among them, T(x,t) is the time surface to be sought, exp is the exponential function, β is the decay rate parameter, which is a constant, and t takes the same value as the grayscale image timestamp t n synchronous;

[0037] Step 7.2, based on the time surface T(x,t), apply the Canny edge detection algorithm to extract edge features and generate an edge mapping image.

[0038] Furthermore, the step 8 comprises the following steps:

[0039] Step 8.1, extracting ORB feature points from the grayscale image and the edge mapping image respectively, and obtaining the corresponding ORB feature points and their descriptor information;

[0040] Step 8.2, performing feature point matching on the grayscale image after feature point detection and the edge mapping image to obtain the coordinates of matched point pairs and the coordinates of unmatched point pairs.

[0041] Furthermore, the step 9 comprises the following steps:

[0042] Step 9.1, calculate the light intensity change rate based on the event stream data obtained in step 2:

[0043] Formula (3)

[0044] Where L represents the rate of change of light intensity, N represents the total number of events, and S i represents the polarity value of the ith event, and △t represents the time span of the event;

[0045] Step 9.2, use nonlinear functions to process the light intensity change rate, linear velocity and angular velocity, and the results are recorded as L ’ 、V ’ , A ’ ;

[0046] Step 9.3, calculate the binocular event camera weight W by the following formula e and the weight W of the stereo grayscale camera g :

[0047] Formula (4)

[0048] Formula (5)

[0049] Among them, α, β, γ, δ, and ζ are the influence coefficients, which are used to balance the effects of different variables;

[0050] Step 9.4, normalize the weights of the stereo event camera and the stereo grayscale camera, and the results are recorded as W e’ and W g ’ , the process is as follows:

[0051] Formula (6)

[0052] Formula (7)

[0053] Step 9.5, use the feature points on the event surface to correct the position of the feature points of the key frame; for the unmatched feature points, use the feature points of the binocular event camera to supplement; for the matched feature points, calculate the fused coordinates by formula (8):

[0054] Formula (8)

[0055] Among them, the coordinates of the feature points of the binocular event camera are P e , the coordinates of the feature points of the grayscale image are P g , the position of the fused feature point is P f ;

[0056] Step 9.6, extract key frames from the fused image feature points.

[0057] Furthermore, the step 11 comprises:

[0058] Step 11.1, respectively calculate the residuals of the inertial sensor pre-integration and pose estimation, that is, the difference between the pre-integration result and the actual measurement value;

[0059] Step 11.2, based on the pre-integrated data of the inertial sensor and the preliminary position and posture of the head-mounted device estimated in step 10, a variable-scale sliding window is constructed, and the variable-scale sliding window dynamically adjusts the window size according to the movement speed of the head-mounted device and the frequency of event occurrence to ensure that only the latest valid data is used for calculation;

[0060] Step 11.3, based on the residual calculated in step 11.1, construct the optimization objective function T and perform joint optimization within the range of the variable scale sliding window:

[0061] Formula (9)

[0062] Among them, r in (t) is the pre-integrated residual of the inertial sensor, which represents the difference between the predicted sensor data and the actual measurement, r pose (t) is the pose estimation residual, which represents the difference between the estimated pose and the observed value;

[0063] Step 11.4, build a three-dimensional local map based on the optimized pose.

[0064] The present invention also provides a dynamic synchronous positioning and mapping system for augmented reality, which is used to implement the dynamic synchronous positioning and mapping method as described above, including a microprocessor, and a binocular event camera, a binocular grayscale camera and an inertial sensor respectively connected to the microprocessor;

[0065] The binocular event camera collects event streams;

[0066] The binocular grayscale camera collects image frame information of the scene;

[0067] The microprocessor includes an event pre-integration module, an event noise removal module, a block motion compensation module, an edge detection module, a feature fusion module and a back-end dynamic optimization module;

[0068] The pre-integration module pre-integrates the inertial sensor data;

[0069] The event noise removal module removes noise from the event stream;

[0070] The block motion compensation module uses the inertial sensor data to perform motion compensation on the edge data;

[0071] The edge detection module converts the event stream into a sparse spatiotemporal representation based on a time surface and extracts edge features to generate an edge mapping image;

[0072] The feature fusion module performs feature point detection on the edge mapping image and the grayscale image respectively, and dynamically fuses the event feature points and the image feature points using the data of the pre-integration and binocular event camera;

[0073] The back-end dynamic optimization module obtains the key frame and preliminary pose information through the fused feature points, and solves the residual term of the pose, and then constructs an optimization function by combining the pre-integration result and the residual term, performs variable-scale joint optimization, further obtains more accurate pose information and constructs a local map, and finally outputs the global pose and global map.

[0074] Compared with the prior art, the present invention has the following beneficial effects:

[0075] In positioning, the present invention uses a binocular grayscale camera to obtain a frame image for ORB feature point detection; uses an edge mapping image to detect ORB feature points; dynamically fuses two feature points to obtain a key frame; obtains the camera posture; constructs an optimization expression based on a pre-integration result and a residual constraint to achieve variable scale joint optimization, obtain the minimum value of the optimization function, and obtain more accurate position information.

[0076] The present invention introduces a high frame rate, low power binocular event camera to build a data acquisition system and method, and then designs a feature fusion module, so that the system can effectively deal with problems such as lighting changes and fast object movement in augmented reality scenes. At the same time, it also takes advantage of the binocular grayscale camera to improve the stability and robustness of the system. Multi-sensor fusion at the feature point level can obtain the camera's posture information more quickly. Compared with the method of separately solving the position of the event camera and the grayscale image and then optimizing it, this method can save a lot of computing resources.

[0077] The present invention integrates a binocular event camera and a binocular grayscale camera to provide a method and system that combines a stereo event stream and a stereo image frame into a synchronous positioning and data acquisition method. The method and system can run in real time under a high-resolution event stream, has richer information, and can adapt to adverse conditions such as low light. The method has great application prospects in the field of augmented reality technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 This is a flow chart of a dynamic synchronous positioning and mapping method for augmented reality of the present invention.

[0079] Figure 2 The present invention is a structural schematic diagram of a head-mounted device in a dynamic synchronous positioning and mapping method for augmented reality when a binocular event camera, a binocular grayscale camera, and an inertial sensor are installed.

[0080] Figure 3 The present invention is a schematic diagram of the structure of a dynamic synchronous positioning and mapping system for augmented reality.

[0081] In the figure: binocular event camera 1; binocular grayscale camera 2; inertial sensor 3. DETAILED DESCRIPTION

[0082] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0083] See also Figure 1 and Figure 2, a dynamic synchronous positioning and mapping method for augmented reality, including: equipping a head-mounted device with a binocular event camera and a binocular grayscale camera, and connecting the binocular event camera, the binocular grayscale camera, the inertial sensor and the microprocessor, the binocular event camera and the binocular grayscale camera work and collect data at the same time; the binocular event camera collects high-temporal resolution event streams, and the binocular grayscale camera is used to collect low-temporal resolution image frame data of the scene; the feature points of the grayscale image are compensated using the high-temporal resolution event stream and pre-integrated data; the key frame information generated after the fusion of feature points is used to match and track the local map, and then jointly optimize with the pre-integrated data to finally generate a complete map and trajectory. Specifically including the following steps:

[0084] Step 1: Determine the status of the binocular event camera, binocular grayscale camera, and inertial sensor, and confirm whether the binocular event camera, binocular grayscale camera, and inertial sensor are connected correctly;

[0085] Step 1 specifically includes: equipping the head-mounted device with a binocular event camera and a binocular grayscale camera, and connecting the binocular event camera, the binocular grayscale camera, the inertial sensor and the microprocessor. After powering on, you can test whether the connection is successful.

[0086] The binocular event camera and the binocular grayscale camera are fixed on the head-mounted device in a rigid body connection manner to ensure that the baseline between the binocular event camera and the binocular grayscale camera is fixed, and the intrinsic and extrinsic parameters of the binocular event camera and the binocular grayscale camera are pre-calibrated.

[0087] The calibrated internal parameters are used to correct the image distortion of the binocular event camera and the binocular grayscale camera, and the calibrated external parameters are used to calculate the position transformation relationship between the images of the binocular event camera and the binocular grayscale camera.

[0088] Among them, image distortion correction is a well-known technology in the field of image correction.

[0089] Step 2: Data collection: To ensure the implementation of the mapping and positioning algorithm, it is necessary to select the binocular event camera and the binocular grayscale camera to work and collect data at the same time.

[0090] Binocular grayscale camera: Use the binocular grayscale camera to obtain the image frame information of the scene, collect images at a fixed frame rate, set the image storage queue, and save the image information. The process is as follows: normalize the original grayscale image data collected by the binocular grayscale camera, then rescale it to [0,255] and store it in the grayscale image queue for map visualization and subsequent processing.

[0091] Binocular event camera: Align the acquisition frame rate of the binocular event camera with that of the binocular grayscale camera. The binocular event camera asynchronously outputs the brightness change of each pixel, that is, the event stream. The data of the event stream includes the brightness change information of each pixel, pixel coordinates, event occurrence timestamp, and event polarity information.

[0092] Aligning the acquisition frame rates of the stereo event camera and the stereo grayscale camera includes the following processes:

[0093] S100, initializing the binocular event camera data acquisition parameters, assuming that the timestamp of the latest grayscale image frame is t n , the timestamp of the next latest frame is t n-1 , set the event queue to record the time t=t between two adjacent frames of the binocular grayscale camera n -t n-1 All event streams within.

[0094] S101, for all event streams acquired within time t, import these event streams into an event queue. If the event queue is full, delete the earliest data in the event queue in turn to release event space.

[0095] Step 3: Synchronize step 2 to collect inertial sensor data and record angular velocity and linear velocity.

[0096] Step 4: Pre-integrate the angular velocity and acceleration data obtained by the inertial sensor to obtain the posture, velocity and position information of the head-mounted device.

[0097] Step 5: Filter the background and hotspot noise in the event stream through the event noise removal module, including the following process:

[0098] Step 5.1, first convert the data stored in the event queue into voxels in three-dimensional space, that is, the smallest segmentation unit in three-dimensional space, and record the number of events in each voxel.

[0099] In step 5.2, the timestamps and spatial coordinates of the events in the voxels are counted and analyzed to remove noise events with unreasonable timestamps, repeated spatial coordinates, and small differences in event timestamps.

[0100] Step 5.3, cluster the events of each voxel according to the spatial position, merge the events in the same spatial voxel, and calculate the time mean and variance of the events in each voxel, use the ratio of the time mean to the variance as the weight factor of the event, and then perform weighted averaging on the events in the voxel, and the result is used as the event output of the voxel.

[0101] Step 5.4, by calculating the time mean and variance of each voxel event output, the event voxel output whose time mean to variance ratio is lower than a preset threshold is removed, thereby obtaining the event stream after removing the noise data.

[0102] Step 6: The block motion compensation module uses the angular velocity and linear velocity provided by the inertial sensor to perform block motion compensation to correct the event stream distortion caused by motion, which specifically includes the following processes:

[0103] Step 6.1, divide the event stream into multiple small blocks by time, with each block being 20-50 milliseconds. Use the angular velocity ω and linear velocity v provided by the inertial sensor to estimate the camera motion within each block, and calculate the transformation matrix R(t) and translation vector T(t) of the block within the block time window.

[0104] Step 6.2, correct the pixel coordinates (u, v) of each event in the block:

[0105] Formula (1)

[0106] Where x is the original event coordinate and x′ is the compensated coordinate.

[0107] Step 7: The edge detection module converts the event stream into a sparse spatiotemporal representation based on a time surface. The event time surface data is characterized by the most recent event timestamp of each pixel. The continuous event stream is encoded into a fixed time distribution graph, edge features are extracted, and an edge map image is generated. The specific process includes the following:

[0108] Step 7.1, represent the event stream as a time surface, traverse the event stream, and find the coordinates of each pixel x=(u,v) T The most recent event within time t, where u is the coordinate of the pixel plane in the x-axis direction, v is the coordinate of the pixel plane in the y-axis direction, and the corresponding timestamp is t last , then t≥t last The time surface at time (x) is defined as:

[0109] Formula (2)

[0110] Among them, T(x,t) is the time surface to be sought, exp is the exponential function, β is the decay rate parameter, which is a constant, and t takes the same value as the grayscale image timestamp t n synchronous.

[0111] Step 7.2: Based on the time surface T(x, t), apply the Canny edge detection algorithm to extract edge features and generate an edge mapping image.

[0112] Step 8, extracting ORB feature points from the grayscale image and the edge mapping image respectively, and matching the feature points of the edge mapping image and the grayscale image, specifically includes the following process:

[0113] Step 8.1, extracting ORB feature points from the grayscale image and the edge mapping image respectively, and obtaining the corresponding ORB feature points and their descriptor information;

[0114] Step 8.2, performing feature point matching on the grayscale image after feature point detection and the edge mapping image to obtain the coordinates of matched point pairs and the coordinates of unmatched point pairs.

[0115] Step 9, combining the event stream collected by the binocular event camera in step 2 and the pre-integrated data in step 6, establish the dynamic weight model of steps 9.1 to 9.5. Extract high-quality feature points from the binocular event camera and the binocular grayscale camera, respectively obtain their spatial position information and local feature description information, i.e., descriptors, by the stereo vision matching algorithm, and fuse this information to construct a key frame feature point set. Specifically, it includes the following processes:

[0116] Step 9.1, calculate the light intensity change rate based on the event stream data obtained in step 2:

[0117] Formula (3)

[0118] Where L represents the rate of change of light intensity, N represents the total number of events, and S i It represents the polarity value of the ith event, and △t represents the time span of the event.

[0119] Step 9.2, use nonlinear functions to process the light intensity change rate, linear velocity and angular velocity, and the results are recorded as L ’ 、V ’ , A ’ .

[0120] Step 9.3, calculate the binocular event camera weight W by the following formula e and the weight W of the stereo grayscale camera g :

[0121] Formula (4)

[0122] Formula (5)

[0123] Among them, α, β, γ, δ, and ζ are the influence coefficients, which are used to balance the effects of different variables.

[0124] Step 9.4, normalize the weights of the stereo event camera and the stereo grayscale camera, and the results are recorded as W e’ and W g ’ , the process is as follows:

[0125] Formula (6)

[0126] Formula (7)

[0127] Step 9.5, use the feature points on the event surface to correct the position of the feature points of the key frame; for the unmatched feature points, use the feature points of the binocular event camera to supplement; for the matched feature points, calculate the fused coordinates by formula (8):

[0128] Formula (8)

[0129] Among them, the coordinates of the feature points of the binocular event camera are P e , the coordinates of the feature points of the grayscale image are P g , the position of the fused feature point is P f .

[0130] Step 9.6, extract key frames from the fused image feature points.

[0131] Step 10: Use the KLT optical flow tracking algorithm on the key frame feature point set to track the motion trajectory of the feature points in adjacent frames. According to the three-dimensional spatial position of the feature point in step 9 and its corresponding two-dimensional image point, use the EPnP algorithm to estimate the initial pose of the head-mounted device, that is, the rotation matrix and translation vector of the head-mounted device, to provide preliminary information for subsequent pose optimization. The initial pose is generally based on the left eye of the grayscale binocular camera, and other cameras can be calculated by the matrix calibrated by external parameters.

[0132] Step 11, combining the pre-integration results of the inertial sensor and the residual terms of the binocular event camera and the binocular grayscale camera, performs variable scale joint optimization to construct a local map, which specifically includes the following processes:

[0133] Step 11.1, respectively calculate the residuals of the inertial sensor pre-integration and pose estimation, that is, the difference between the pre-integration result and the actual measurement value;

[0134] Step 11.2, based on the pre-integrated data of the inertial sensor and the preliminary position and posture of the head-mounted device estimated in step 10, a variable-scale sliding window is constructed, and the variable-scale sliding window dynamically adjusts the window size according to the movement speed of the head-mounted device and the frequency of event occurrence to ensure that only the latest valid data is used for calculation;

[0135] Step 11.3, construct the optimization objective function based on the residual calculated in step 11.1 , and perform joint optimization over the range of a sliding window of varying scale:

[0136] Formula (9)

[0137] Among them, r in (t) is the pre-integrated residual of the inertial sensor, which represents the difference between the predicted sensor data and the actual measurement, r pose (t) is the pose estimation residual, which represents the difference between the estimated pose and the observed value.

[0138] Step 11.4, build a three-dimensional local map based on the optimized pose.

[0139] Step 12: Output accurate global pose and global map.

[0140] It should be noted that the above-mentioned stereo vision matching algorithm, EPnP algorithm, inertial sensor pre-integration, KLT optical flow tracking algorithm, ORB feature point extraction algorithm, and Canny edge detection algorithm are all well-known technologies in the field of information technology and will not be elaborated on.

[0141] See also Figure 3 The present invention also provides a dynamic synchronous positioning and mapping system for augmented reality, which is used to implement the dynamic synchronous positioning and mapping method as described above, including a microprocessor, and a binocular event camera, a binocular grayscale camera and an inertial sensor electrically connected to the microprocessor respectively. The binocular event camera collects event streams. The binocular grayscale camera collects image frame information of the scene. The microprocessor includes a pre-integration module, an event noise removal module, a block motion compensation module, an edge detection module, a feature fusion module and a back-end dynamic optimization module.

[0142] The pre-integration module performs pre-integration on the inertial sensor data. The pre-integration module is a computer program, and its specific implementation can be found in the above step 4, which will not be described in detail here.

[0143] The event noise removal module removes noise from the event stream. The event noise removal module is a computer program, and its specific implementation can be found in the above step 5, which will not be described here.

[0144] The block motion compensation module uses the inertial sensor data to perform motion compensation on the edge data. The block motion compensation module is a computer program, and its specific implementation can be found in the above step 6, which will not be described in detail here.

[0145] The edge detection module converts the event stream into a sparse spatiotemporal representation based on the time surface, extracts edge features, and then generates an edge mapping image. The edge detection module is a computer program, and its specific implementation can be found in step 7 above, which will not be repeated here.

[0146] Among them, the feature fusion module performs feature point detection on the edge mapping image and the grayscale image respectively, and dynamically fuses the event feature points and the image feature points using the data of the pre-integration and binocular event camera. The feature fusion module is a computer program, and its specific implementation can be found in the above step 8, which will not be repeated here.

[0147] Among them, the back-end dynamic optimization module obtains the key frame and preliminary pose information through the fused feature points, and solves the residual term of the pose, and then combines the result of the pre-integration and the residual term to construct an optimization function, performs variable-scale joint optimization, further obtains more accurate pose information and constructs a local map, and finally outputs the global pose and global map. The back-end dynamic optimization module is a computer program, and its specific implementation can be found in the above steps 9 to 12, which will not be repeated here.

[0148] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A dynamic synchronous positioning and mapping method for augmented reality, characterized in that: The following steps are involved: Step 1: calibrate the internal and external parameters of the binocular event camera and the binocular grayscale camera, and perform image distortion correction; Step 2: Use the binocular event camera and the binocular grayscale camera to collect data at the same time, obtain image frames and event streams, and perform time synchronization to ensure data consistency. The binocular grayscale camera saves the grayscale image. Step 3, using inertial sensors to collect angular velocity and linear velocity; Step 4, pre-integrate the angular velocity and acceleration data obtained by the inertial sensor to obtain the posture, velocity and position information of the head mounted device; Step 5: Remove background and hotspot noise from the event stream; Step 6: Use the angular velocity and linear velocity provided by the inertial sensor to perform block motion compensation and correct event stream distortion; Step 7, convert the event stream into a time surface representation, extract edge features and generate an edge map image; Step 8, extracting feature points from the grayscale image and the edge mapping image respectively, and matching them; Step 9, using the dynamic weight model to fuse the feature point data of the event camera and the grayscale camera to construct a feature point set of the key frame, including the following steps: Step 9.1, calculate the light intensity change rate based on the event stream data obtained in step 2: Formula (3) Where L represents the rate of change of light intensity, N represents the total number of events, S i represents the polarity value of the ith event, and △t represents the time span of the event; Step 9.2, use nonlinear functions to process the light intensity change rate, linear velocity and angular velocity, and the results are recorded as L ’ 、V ’ , A ’ ; Step 9.3, calculate the binocular event camera weight W by the following formula e and the weight W of the stereo grayscale camera g : Formula (4) Formula (5) Among them, α, β, γ, δ, and ζ are the influence coefficients, which are used to balance the effects of different variables; Step 9.4, normalize the weights of the stereo event camera and the stereo grayscale camera, and the results are recorded as W e ’ and W g ’ , the process is as follows: Formula(6) Formulas (7) Step 9.5, use the feature points on the event surface to correct the position of the feature points of the key frame; for the unmatched feature points, use the feature points of the binocular event camera to supplement; for the matched feature points, calculate the fused coordinates by formula (8): Formulas (8) Among them, the coordinates of the feature points of the binocular event camera are P e , the coordinates of the feature points of the grayscale image are P g , the position of the fused feature point is P f ; Step 9.6, extract key frames from the fused image feature points; Step 10, using the KLT optical flow tracking algorithm to track the motion trajectory of the feature points of the key frame; using the EPnP algorithm to estimate the initial posture of the head-mounted device to provide preliminary information for subsequent posture optimization; Step 11, combining the pre-integrated data and the preliminary pose data, performing variable-scale joint optimization, and constructing a local map; Step 12: Output the global pose and global map.

2. The method for dynamic synchronous positioning and mapping for augmented reality according to claim 1, characterized in that: In step 2, the acquisition time of the binocular event camera and the binocular grayscale camera is aligned The process includes: S100, initializing the binocular event camera data acquisition parameters, assuming that the timestamp of the latest grayscale image frame is t n , the timestamp of the next latest frame is t n-1 , set the event queue to record the time t=t between two adjacent frames of the binocular grayscale camera n -t n-1 All event streams within; S101, for all event streams acquired within time t, import these event streams into an event queue. If the event queue is full, delete the earliest data in the event queue in turn to release event space.

3. The method for dynamic synchronous positioning and mapping for augmented reality according to claim 2, characterized in that: The step 5 comprises the following steps: Step 5.1, converting the data stored in the event queue into voxels in three-dimensional space, and recording the number of events in each voxel; Step 5.2, count and analyze the timestamps and spatial coordinates of the events in the voxels, and remove noise events with unreasonable timestamps, repeated spatial coordinates, and small differences in event timestamps; Step 5.3, cluster the events of each voxel according to the spatial position, merge the events in the same spatial voxel, and calculate the time mean and variance of the events in each voxel. Use the ratio of the time mean to the variance as the weight factor of the event, so as to perform weighted averaging on the events in the voxel, and the result is used as the event output of the voxel; In step 5.4, by calculating the time mean and variance of each voxel event output, the event voxel output with a small ratio of the time mean to the variance is removed, thereby obtaining the event stream after removing the noise data.

4. The method for dynamic synchronous positioning and mapping for augmented reality according to claim 1, characterized in that: The step 6 comprises the following steps: Step 6.1, divide the event stream into multiple small blocks by time, with each block being 20-50 milliseconds. Use the angular velocity ω and linear velocity v provided by the inertial sensor to estimate the camera motion within each block, and calculate the transformation matrix R(t) and translation vector T(t) of the block within the block time window. Step 6.2, correct the pixel coordinates (u, v) of each event in the block: Formula (1) Where x is the original event coordinate and x′ is the compensated coordinate.

5. The method for dynamic synchronous positioning and mapping for augmented reality according to claim 1, characterized in that: The step 7 comprises the following steps: Step 7.1, represent the event stream as a time surface, traverse the event stream, and find the coordinates of each pixel x=(u,v) T The most recent event within time t, where u is the coordinate of the pixel plane in the x-axis direction, v is the coordinate of the pixel plane in the y-axis direction, and the corresponding timestamp is t last , then t≥t last The time surface at time (x) is defined as: Formula (2) Among them, T(x,t) is the time surface to be sought, exp is the exponential function, β is the decay rate parameter, which is a constant, and t takes the same value as the grayscale image timestamp t n synchronous; Step 7.2, based on the time surface T(x,t), apply the Canny edge detection algorithm to extract edge features and generate an edge mapping image.

6. The method for dynamic synchronous positioning and mapping for augmented reality according to claim 1, characterized in that: The step 8 comprises the following steps: Step 8.1, extracting ORB feature points from the grayscale image and the edge mapping image respectively, and obtaining the corresponding ORB feature points and their descriptor information; Step 8.2, performing feature point matching on the grayscale image after feature point detection and the edge mapping image to obtain the coordinates of matched point pairs and the coordinates of unmatched point pairs.

7. The method for dynamic synchronous positioning and mapping for augmented reality according to claim 1, characterized in that: The step 11 comprises: Step 11.1, respectively calculate the residuals of the inertial sensor pre-integration and pose estimation, that is, the difference between the pre-integration result and the actual measurement value; Step 11.2, based on the pre-integrated data of the inertial sensor and the preliminary position and posture of the head-mounted device estimated in step 10, a variable-scale sliding window is constructed, and the variable-scale sliding window dynamically adjusts the window size according to the movement speed of the head-mounted device and the frequency of event occurrence to ensure that only the latest valid data is used for calculation; Step 11.3, based on the residual calculated in step 11.1, construct the optimization objective function T and perform joint optimization within the range of the variable scale sliding window: Formulas (9) Among them, r in (t) is the pre-integrated residual of the inertial sensor, which represents the difference between the predicted sensor data and the actual measurement, r pose (t) is the pose estimation residual, which represents the difference between the estimated pose and the observed value; Step 11.4, build a three-dimensional local map based on the optimized pose.

8. A dynamic synchronous positioning and mapping system for augmented reality, used to implement the dynamic synchronous positioning and mapping method as described in any one of claims 1 to 7, characterized in that: It includes a microprocessor, and a binocular event camera, a binocular grayscale camera and an inertial sensor respectively connected to the microprocessor; The binocular event camera collects event stream; The binocular grayscale camera collects image frame information of the scene; The microprocessor includes an event pre-integration module, an event noise removal module, a block motion compensation module, an edge detection module, a feature fusion module and a back-end dynamic optimization module; The pre-integration module pre-integrates the inertial sensor data; The event noise removal module removes noise from the event stream; The block motion compensation module uses the inertial sensor data to perform motion compensation on the edge data; The edge detection module converts the event stream into a sparse spatiotemporal representation based on a time surface and extracts edge features to generate an edge mapping image; The feature fusion module performs feature point detection on the edge mapping image and the grayscale image respectively, and dynamically fuses the event feature points and the image feature points using the data of the pre-integration and binocular event camera; The back-end dynamic optimization module obtains the key frame and preliminary pose information through the fused feature points, and solves the residual term of the pose, and then constructs an optimization function by combining the pre-integration result and the residual term, performs variable-scale joint optimization, further obtains more accurate pose information and constructs a local map, and finally outputs the global pose and global map.

Images