Multi-sensor latency estimation method, apparatus, xr device, and storage medium

Abstract

The application provides a multi-sensor time delay estimation method and device, an XR device and a storage medium, and relates to the technical field of computer vision. The method comprises the following steps: acquiring image data of a vision sensor, determining a parameterized coordinate of a three-dimensional space point in a space and a change rate of a direction vector in the parameterized coordinate according to the image data; correcting the parameterized coordinate of the three-dimensional space point according to the change rate of the direction vector and a multi-sensor time delay parameter; calculating a re-projection residual of the corrected parameterized coordinate of the three-dimensional space point projected to a target image frame to construct an optimization problem; and solving the optimization problem to obtain the multi-sensor time delay parameter. The observation data from different sensors can be aligned in time, and the result accuracy of state estimation is improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a multi-sensor time delay estimation method, apparatus, XR device, and storage medium. Background Technology

[0002] When performing state estimation, existing smart devices primarily acquire state information such as the device's position, velocity, and attitude through a combination of multiple sensors. For example, they obtain the device's state information using Simultaneous Localization and Mapping (SLAM) technology.

[0003] However, sensors often face delays in data acquisition and transmission during the data collection process. For example, in visual inertial odometry (VIO), because there are delays in exposure time and data transmission during the image acquisition process of the camera, the timestamp of the image data acquired by the camera deviates from the actual time when the camera shutter opens for exposure, resulting in the timestamp of the image data acquired by the camera not corresponding to the actual time when the physical state of the carrier occurs.

[0004] When performing state estimation of smart devices based on multi-sensor fusion positioning, this timestamp drift introduces systematic state estimation errors, causing observation data from different sensors to be out of sync in time, thus reducing the accuracy of the state estimation results. Summary of the Invention

[0005] This invention provides a multi-sensor time delay estimation method, apparatus, XR device, and storage medium to address the shortcomings of existing technologies where observation data from different sensors cannot be aligned in time, thereby improving the accuracy of state estimation results.

[0006] This invention provides a multi-sensor time delay estimation method, comprising: acquiring image data from a visual sensor, and determining, based on the image data, the parameterized coordinates of a three-dimensional spatial point in space and the rate of change of the direction vector in the parameterized coordinates;

[0007] The parameterized coordinates of the three-dimensional spatial point are corrected based on the rate of change of the direction vector and the multi-sensor delay parameters.

[0008] The reprojection residuals of the corrected parameterized coordinates of the three-dimensional spatial points onto the target image frame are calculated to construct an optimization problem;

[0009] The multi-sensor time delay parameters are obtained by solving the optimization problem.

[0010] According to a multi-sensor time delay estimation method provided by the present invention, the step of determining the parametric coordinates of a three-dimensional spatial point in space based on the image data includes:

[0011] The feature points of the current image frame are determined based on the image data, and corresponding three-dimensional spatial points are generated using the feature points.

[0012] The three-dimensional spatial points are parameterized according to the projection model to determine the parameterized coordinates of the three-dimensional spatial points in space.

[0013] According to a multi-sensor time delay estimation method provided by the present invention, determining the rate of change of the direction vector in the parameterized coordinates based on the image data includes:

[0014] Based on the image data, the feature points of the current image frame are determined, and the planar feature pixel velocity of the feature points is calculated;

[0015] Based on the projection relationship, the planar feature pixel velocity of the feature point is converted into the rate of change of the direction vector of the corresponding three-dimensional spatial point.

[0016] According to a multi-sensor time delay estimation method provided by the present invention, the calculation of the planar feature pixel velocity of the feature point includes:

[0017] Based on the feature points of the current image frame, obtain the same feature points in the reference image frame;

[0018] The first coordinates of the same feature point in the reference image frame, the second coordinates of the feature point in the current image frame, and the time difference between the reference image frame and the current image frame are obtained to calculate the planar feature pixel velocity of the feature point.

[0019] According to a multi-sensor time delay estimation method provided by the present invention, the planar feature pixel velocity of the feature point is converted into the rate of change of the direction vector of the corresponding three-dimensional spatial point based on the projection relationship, including:

[0020] According to the chain rule, the projection relationship from a three-dimensional spatial point to a pixel coordinate system is converted into a correspondence between the planar feature pixel velocity of the feature point and the rate of change of the direction vector of the corresponding three-dimensional spatial point.

[0021] Based on the correspondence, the planar feature pixel velocity of the feature point is converted into the rate of change of the direction vector of the corresponding three-dimensional spatial point.

[0022] According to a multi-sensor time delay estimation method provided by the present invention, the reprojection residual of the corrected parameterized coordinates of the three-dimensional spatial points projected onto the target image frame is calculated, including:

[0023] The corrected three-dimensional spatial points are obtained by back-projecting the parameterized coordinates of the corrected three-dimensional spatial points, and then the corrected three-dimensional spatial points are projected onto the target image frame using the camera pose.

[0024] The corrected 3D spatial points are processed by the inverse process of back projection to obtain the parameterized coordinates of the corrected 3D spatial points in the target image frame, so as to calculate the reprojection residual.

[0025] According to a multi-sensor time delay estimation method provided by the present invention, the reprojection residual is calculated, including:

[0026] Determine the parameterized coordinates of the three-dimensional spatial points corresponding to the feature points of the target image frame, as well as the rate of change of the direction vector in the parameterized coordinates;

[0027] The actual parameterized coordinates of the three-dimensional spatial point are corrected based on the rate of change of the direction vector and the multi-sensor delay parameters.

[0028] The reprojection residual of the feature point is calculated using the corrected actual parameterized coordinates and the corrected parameterized coordinates of the three-dimensional spatial point in the target image frame.

[0029] The present invention also provides a multi-sensor time delay estimation device, comprising:

[0030] The parameterized data determination module is used to acquire image data from a vision sensor, and to determine the parameterized coordinates of a three-dimensional spatial point in space and the rate of change of the direction vector in the parameterized coordinates based on the image data.

[0031] The parameterized data correction module is used to correct the parameterized coordinates of the three-dimensional spatial points based on the rate of change of the direction vector and the multi-sensor delay parameters.

[0032] The optimization problem construction module calculates the reprojection residual of the corrected parameterized coordinates of the three-dimensional space points projected onto the target image frame to construct the optimization problem;

[0033] The time delay parameter solving module is used to solve the optimization problem to obtain the multi-sensor time delay parameters.

[0034] The present invention also provides an XR device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-sensor delay estimation method as described above.

[0035] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-sensor delay estimation method as described above.

[0036] The multi-sensor time delay estimation method, apparatus, XR device, and storage medium provided by this invention acquire image data from a visual sensor, determine the parametric coordinates of a three-dimensional spatial point and the rate of change of the direction vector in the parametric coordinates based on the image data, correct the parametric coordinates of the three-dimensional spatial point based on the rate of change of the direction vector and the multi-sensor time delay parameters, incorporate the multi-sensor time delay parameters into the corrected parametric coordinates of the three-dimensional spatial point, and calculate the reprojection residual of the corrected parametric coordinates of the three-dimensional spatial point onto the target image frame. This allows the multi-sensor time delay parameters to be incorporated into the reprojection residual. An optimization problem is constructed based on the reprojection residual with the multi-sensor time delay parameters incorporated, and the multi-sensor time delay parameters are obtained by solving the optimization problem. This enables temporal alignment of observation data from different sensors, solves the systematic error in multimodal data state estimation caused by timestamp drift, and improves the accuracy of state estimation results. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0038] Figure 1 This is an exemplary system architecture diagram in which some embodiments of the present invention can be applied.

[0039] Figure 2 This is a flowchart illustrating the multi-sensor time delay estimation method provided by the present invention.

[0040] Figure 3 This is a schematic diagram illustrating an application scenario of the multi-sensor time delay estimation method provided by the present invention.

[0041] Figure 4 This is a schematic diagram of the framework of the multi-sensor time delay estimation method provided by the present invention.

[0042] Figure 5 This is a schematic diagram of the multi-sensor time delay estimation device provided by the present invention.

[0043] Figure 6 This is a schematic diagram of the structure of the XR device provided by the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0045] Currently, when using multiple sensors to estimate the state of smart devices and obtain information such as the position, speed, and attitude of the smart device's carrier, relying on simple hardware synchronization or fixed time delay to synchronize the data collected by multiple sensors often results in the observation data from different sensors not being effectively aligned in time, and the data from different sensors failing to reflect the physical state of the carrier at the same moment.

[0046] State estimation refers to the process of calculating the state of a carrier based on sensor measurements using certain algorithmic logic. State estimation generally includes position calculation, velocity calculation, and attitude calculation. Time synchronization aligns the data collected by different sensors under the same time reference to ensure temporal consistency during fusion.

[0047] In view of this, embodiments of the present invention provide a multi-sensor delay estimation method, apparatus, XR device, and storage medium. The technical solutions of the embodiments of the present invention will be described below with reference to the accompanying drawings.

[0048] Figure 1 An exemplary system architecture 100 is shown, in which embodiments of the multi-sensor delay estimation method or multi-sensor delay estimation apparatus of the present invention can be applied.

[0049] like Figure 1 As shown, the system architecture 100 may include a vision sensor 101, an inertial measurement unit 102, and a lidar 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the vision sensor 101, the inertial measurement unit 102, the lidar 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables.

[0050] The inertial measurement unit 102 and the lidar 103 are exemplary sensors. The sensors can also be Global Positioning System (GPS) and wheeled odometers, etc., without specific limitations.

[0051] Server 105 can be a remote server providing various services, such as a remote server that analyzes and processes data from the vision sensor 101, inertial measurement unit 102, and lidar 103. It can also be a local computing device on the target carrier side, such as the local computing device of an XR device configured on the target carrier, or a local computing device on a robot or autonomous vehicle, etc., without specific limitations. Thus, the target carrier state estimation method provided in this embodiment of the invention can be executed by a remote server or by a local computing device on the target carrier side.

[0052] It should be understood that Figure 1 The number of sensors, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of sensors, networks, and servers can be included.

[0053] Figure 2 This is a flowchart illustrating the multi-sensor time delay estimation method provided by the present invention, as shown below. Figure 2 As shown, the method includes the following steps.

[0054] Step 201: Acquire image data from a vision sensor to determine the parameterized coordinates of a three-dimensional spatial point in space and the rate of change of the direction vector in the parameterized coordinates based on the image data.

[0055] For example, the visual sensor can be a stereo camera of a visual inertial odometry system or a fisheye camera of a surround-view system.

[0056] It should be noted that the image data sent by the vision sensor can be acquired, feature detection can be performed on the image data, and the parameterized coordinates of the three-dimensional spatial points in the space corresponding to the reference image frame and the rate of change of the direction vector in the parameterized coordinates can be determined based on the result of the feature detection.

[0057] In this context, a 3D spatial point is a point with a definite location in three-dimensional space. Parametric coordinates refer to the way a 3D spatial point is represented using mathematical methods or models. A direction vector is a unit vector pointing from the camera center of the vision sensor to the 3D spatial point, used to represent the direction from the camera origin to the 3D spatial point.

[0058] Delay estimation is the process of adjusting the timestamps of the data collected by each sensor so that the data collected by each sensor is aligned with the physical state of the carrier at the same moment.

[0059] Step 202: Correct the parameterized coordinates of the three-dimensional spatial point according to the rate of change of the direction vector and the multi-sensor delay parameters.

[0060] Among them, the multi-sensor delay parameter is an indicator for analyzing and compensating for the time difference in data acquired by visual sensors when multiple sensors are combined for state estimation.

[0061] It should be noted that the estimated change can be determined based on the rate of change of the direction vector and the multi-sensor delay parameters. The parameterized coordinates of the three-dimensional spatial point can be corrected based on the estimated change to obtain the actual parameterized coordinates of the three-dimensional spatial point in the current image frame corresponding to the image data.

[0062] For example, the corrected parameterized coordinates of a three-dimensional spatial point can be obtained by subtracting the estimated change from the parameterized coordinates of the three-dimensional spatial point. Alternatively, the weight can be determined based on the estimated change and a preset correspondence, and the corrected parameterized coordinates of the three-dimensional spatial point can be obtained by adjusting the parameterized coordinates of the three-dimensional spatial point according to the weight. This invention does not limit the scope of the invention.

[0063] Step 203: Calculate the reprojection residual of the corrected parameterized coordinates of the three-dimensional spatial points projected onto the target image frame to construct an optimization problem.

[0064] The reprojection residual is an index that measures the error between a 3D point in a reference image frame and its actual observed point in the target image frame after projection onto the target image frame. The optimization problem is a mathematical task aimed at minimizing the reprojection residual.

[0065] By minimizing the sum of squares of the reprojection residuals, the delay parameters are jointly optimized. This allows for the time alignment of multi-sensor data with system state variables (such as pose and spatial point parameters).

[0066] It is understandable that by projecting the parameterized coordinates of the three-dimensional spatial points of the reference image frame onto the target image frame, it is easy to compare the actual observed feature points in the target image frame with the predicted feature points in the projected target image frame to obtain the reprojection residual. Since the parameterized coordinates of the three-dimensional spatial points of the reference image frame are obtained by adjusting the multi-sensor time delay parameters, the impact of the time delay parameters on the positional consistency of the feature points can be evaluated based on the optimization problem constructed from the reprojection residual.

[0067] Step 204: Solve the optimization problem to obtain the multi-sensor time delay parameters.

[0068] It should be noted that there are many ways to solve optimization problems, such as sparse matrix factorization, Gauss-Newton method, etc., and this invention does not limit the methods.

[0069] It is understandable that when performing state estimation by combining multiple sensors, it is necessary to acquire image data from the visual sensor and data collected by other sensors at the same time. However, the acquisition and transmission of data by the sensors take time, which may result in inconsistent physical states of the carrier represented by the data collected by different sensors.

[0070] To align data acquired by different sensors (including image data from visual sensors) to the same moment, ensuring that the same object observed by different sensors corresponds spatially, current methods typically correct the timestamps of each sensor's data individually. However, the multi-sensor time delay estimation method provided in this invention acquires image data from a visual sensor to determine the parametric coordinates of a 3D spatial point and the rate of change of the direction vector within those coordinates. Based on the rate of change of the direction vector and multi-sensor time delay parameters, the method corrects the parametric coordinates of the 3D spatial point, incorporating these parameters into the corrected coordinates. Furthermore, it calculates the reprojection residual of the corrected 3D spatial point's parametric coordinates onto the target image frame, thus incorporating the multi-sensor time delay parameters into the reprojection residual. An optimization problem is then constructed based on this reprojection residual, and solving the optimization problem yields the multi-sensor time delay parameters. This method aligns observation data from different sensors in time, resolves the systematic error in multimodal data state estimation caused by timestamp drift, and improves the accuracy of state estimation results.

[0071] The carrier for physical state assessment is a movable entity, such as a person, robot, or various types of vehicles. The carrier can be directly equipped with various sensors; for example, cameras and LiDAR can be directly mounted on autonomous vehicles. Sensors can also be integrated through smart devices, such as wearable devices on the human body. By configuring various sensors, image frames and motion state parameters can be collected to estimate the state of the target carrier. These wearable devices can be XR devices, smart accessories, etc. XR devices are a general term for VR, AR, and MR devices, specifically including smart glasses and virtual reality head-mounted displays (VR headsets).

[0072] Specifically, the multi-sensor time delay estimation method provided in this embodiment of the invention can be applied to, for example... Figure 3 In the application scenario shown. Figure 3 In this application scenario, the carrier of the smart device is a person. It collects image data and IMU data by configuring smart glasses 301. The smart glasses 301 are equipped with a visual sensor 302 and an auxiliary sensor 303. The smart glasses 301 can be connected to a remote server 305 via a network 304. In this case, the multi-sensor latency estimation method provided in this embodiment of the invention can be executed by the remote server or by a local computing device on the carrier side of the smart device.

[0073] Based on the above embodiments, determining the parameterized coordinates of a three-dimensional spatial point in space according to the image data includes: determining feature points of the current image frame according to the image data, and generating corresponding three-dimensional spatial points through the feature points; parameterizing the three-dimensional spatial points according to the projection model to determine the parameterized coordinates of the three-dimensional spatial points in space.

[0074] It should be noted that there are many ways to perform feature detection on image data to determine the feature points of the current image frame, such as Harris corner detection, scale-invariant feature transform (SIFT), etc., and this invention does not limit the methods.

[0075] There are many ways to process feature points to generate corresponding three-dimensional spatial points, such as through triangulation or convolutional neural network generation, and this invention does not limit these methods.

[0076] For example, the projection model can be a stereographic projection model (also known as spherical projection, stereographic projection, etc.). Based on the projection model, a 3D spatial point is parameterized to obtain the direction vector and inverse depth of the 3D spatial point projected onto the reference frame. The parameterized coordinates of the 3D spatial point can include three degrees of freedom: the first two degrees of freedom represent the direction vector of the 3D spatial point in the reference frame, and the last degree of freedom represents the inverse depth of the 3D spatial point in the reference frame.

[0077] Among them, the Stereographic projection model provides a projection method that can project three-dimensional spatial points onto a unit sphere under the camera frame, parameterize the coordinates of three-dimensional spatial points into a combination of the direction vector and depth of the sphere, and provide a compact representation of the coordinates of three-dimensional spatial points.

[0078] Understandably, using a projection model to parametrically represent points in three-dimensional space can simplify the mathematical representation of these points, reduce the complexity of subsequent calculations, and facilitate the efficient processing of a large number of feature points during the optimization process.

[0079] Based on any of the above embodiments, determining the rate of change of the direction vector in the parameterized coordinates according to the image data includes: determining the feature points of the current image frame according to the image data, and calculating the planar feature pixel velocity of the feature points;

[0080] Based on the projection relationship, the planar feature pixel velocity of the feature point is converted into the rate of change of the direction vector of the corresponding three-dimensional spatial point.

[0081] Among them, the projection relationship refers to the mapping relationship between the parameterized coordinates of a 3D spatial point and the planar feature pixel coordinates of the feature point, with the 3D spatial point corresponding to the feature point.

[0082] For example, the projection relationship can be:

[0083]

[0084] in, These are the planar feature pixel coordinates of the feature points. These are the parameterized coordinates of a point in three-dimensional space. This describes the process of generating 3D spatial points by back-projection based on the spatial point orientation vector and inverse depth in parametric coordinates. This represents the process of projecting a point in three-dimensional space onto the pixel coordinate system.

[0085] Here, u and v can be coordinates obtained by mapping direction vectors onto the projection plane through spherical projection, and inv_d is the inverse depth, which is the reciprocal of the depth. The projection plane can be defined on a normalized plane, and the depth is the distance between a point in 3D space and the center of the camera.

[0086] It should be noted that there are many ways to project spatial points onto a pixel coordinate system, such as camera projection models and multi-sensor time delay estimation, and this invention does not limit these methods.

[0087] Understandably, compared to existing methods that model directly on the pixel plane, combining projection relationships to convert the velocity of planar feature pixels into the rate of change of the direction vector can eliminate the nonlinear error caused by the coupling between depth and focal length, and improve the geometric consistency and accuracy of time delay estimation.

[0088] Specifically, in some embodiments, converting the planar feature pixel velocity of the feature point into the rate of change of the direction vector of the corresponding three-dimensional spatial point according to the projection relationship includes: converting the projection relationship from the three-dimensional spatial point to the pixel coordinate system into a correspondence between the planar feature pixel velocity of the feature point and the rate of change of the direction vector of the corresponding three-dimensional spatial point according to the chain rule; and converting the planar feature pixel velocity of the feature point into the rate of change of the direction vector of the corresponding three-dimensional spatial point according to the correspondence.

[0089] Among them, the rate of change of the direction vector of a three-dimensional spatial point is used to describe the speed at which the direction vector changes over time, and can represent how the three-dimensional spatial point rotates in space from the camera's line of sight.

[0090] For example, the aforementioned projection relationship can be transformed to obtain the following formula:

[0091]

[0092] Understandably, through It can establish the correspondence between the planar feature pixel velocity of a feature point and the rate of change of the direction vector of the corresponding 3D spatial point, thereby determining the rotation rate of the 3D spatial point from the camera's line of sight based on the planar feature pixel velocity of the feature point, providing a foundation for subsequent optimization problems.

[0093] Understandably, compared to the direct conversion method based solely on geometric relationships, converting the projection relationship from three-dimensional spatial points to the pixel coordinate system into the correspondence between the planar feature pixel velocity and its corresponding three-dimensional spatial direction vector change rate according to the chain rule can maintain the continuity and differentiability of the projection mapping, improve the accuracy and stability of the direction vector change rate calculation, and achieve accurate analytical mapping from pixel motion to direction vector change.

[0094] Based on any of the above embodiments, the step of calculating the planar feature pixel velocity of the feature point includes: obtaining the same feature point in the reference image frame based on the feature point of the current image frame; obtaining the first coordinate of the same feature point in the reference image frame, the second coordinate of the feature point in the current image frame, and the time difference between the reference image frame and the current image frame, so as to calculate the planar feature pixel velocity of the feature point.

[0095] Among them, the same feature points, also known as the same feature points or the same point, refer to the corresponding points of the same object point in different image frames.

[0096] In some embodiments, feature points obtained by feature detection of image data include keypoints and descriptors. Keypoints are used to describe the coordinates of the feature points. A descriptor is a high-dimensional vector used to describe image information surrounding the feature points.

[0097] The system can obtain identical feature points in a reference image frame based on the feature points and descriptors of the current image frame. It can also obtain the first coordinates of the identical feature points in the reference image frame and the second coordinates of the identical feature points in the current image frame based on key points. The time difference between the reference and current image frames can be determined using the timestamps of both frames. The planar feature pixel distance between identical feature points is determined based on the first and second coordinates. Finally, the planar feature pixel velocity of the corresponding object point is calculated based on the planar feature pixel distance and the time difference.

[0098] It is understandable that by backprojecting the corrected 3D spatial points and reprojecting them onto the target image frame using the camera pose, and then obtaining the parameterized coordinates of the corrected feature points in the target image frame through the inverse process of backprojection, a consistent geometric transformation of backprojection, inter-frame projection, and parameterization is formed, thereby improving the accuracy of residual calculation.

[0099] Based on any of the above embodiments, calculating the reprojection residual of the corrected parametric coordinates of the three-dimensional spatial point projected onto the target image frame includes: backprojecting the corrected parametric coordinates of the three-dimensional spatial point to obtain the corrected three-dimensional spatial point, and projecting the corrected three-dimensional spatial point onto the target image frame using the camera pose; processing the corrected three-dimensional spatial point through the inverse process of backprojection to obtain the corrected parametric coordinates of the three-dimensional spatial point in the target image frame, and calculating the reprojection residual.

[0100] It should be noted that in the multi-sensor time delay estimation based on vision, the reprojection residual of the corresponding point can be calculated by tracking the projection of the corresponding point in different image frames for optimization.

[0101] The target image frame refers to the image frame in which the same point is observed later, while the reference image frame refers to the image frame in which the same point is observed earlier.

[0102] It is understandable that when estimating the time delay of multiple sensors to achieve time synchronization of multiple sensors, after correcting the parameterized coordinates of the three-dimensional spatial points by the time delay parameters of multiple sensors, the parameterized coordinates of the three-dimensional spatial points are restored to the three-dimensional spatial points by back projection. This enables the three-dimensional spatial points to be in the same coordinate system as the camera pose, so that the corrected three-dimensional spatial points are projected onto the target image frame to obtain the predicted direction vector of the corresponding points in the target image frame.

[0103] It should be noted that the actual direction vector of the corresponding point in the target image frame can also be called the observation vector. The reprojection residual can be calculated and optimized by using the observation vector and the predicted direction vector of the corresponding point in the target image frame.

[0104] Specifically, in some embodiments, calculating the reprojection residual includes: determining the parameterized coordinates of the three-dimensional spatial point corresponding to the feature point of the target image frame and the rate of change of the direction vector in the parameterized coordinates; correcting the actual parameterized coordinates of the three-dimensional spatial point according to the rate of change of the direction vector and the multi-sensor delay parameter; and calculating the reprojection residual of the feature point using the corrected actual parameterized coordinates and the corrected parameterized coordinates of the three-dimensional spatial point in the target image frame.

[0105] It should be noted that in this embodiment, the current image frame of the image data from the vision sensor is used as the target image frame. Based on this, a reference image frame that was acquired before the target image frame and observed the same point can be used as the reference image frame, or the image frame that first observed the same point can be used as the reference image frame, etc. This embodiment does not limit this.

[0106] The method for correcting the actual parametric coordinates of the three-dimensional spatial points of the target image frame is basically the same as the method for correcting the parametric coordinates of the three-dimensional spatial points mentioned above, and will not be repeated here.

[0107] For example, the reprojection residual can be calculated using the following formula:

[0108]

[0109] in, It is the reprojection residual of the i-th point in three-dimensional space. It is the direction vector of the i-th three-dimensional spatial point in the target image frame. It is the rate of change of the direction vector of the i-th three-dimensional spatial point in the target image frame. It is a multi-sensor delay parameter. express The reverse process of the operation, This represents the relative pose transformation matrix from the camera coordinate system of the reference image frame to the camera coordinate system of the target image frame. It is the direction vector of the i-th three-dimensional point in the reference image frame. It is the rate of change of the direction vector of the i-th three-dimensional spatial point in the reference image frame.

[0110] Understandably, constructing an optimization problem by explicitly incorporating multi-sensor time delay parameters when building reprojection residuals can define observation residuals of different modes on the same time base, align observation data from different sensors in time, solve the systematic error in state estimation of multi-modal data, and improve the accuracy of state estimation results.

[0111] Meanwhile, by explicitly incorporating multi-sensor time delay parameters when constructing reprojection residuals, the dynamic changes in the orientation of three-dimensional spatial points in space can be captured during the optimization process, further reducing the systematic error of multimodal data state estimation.

[0112] Figure 4 This is a schematic diagram of the architecture of the multi-sensor time delay estimation method provided by the present invention, as shown below. Figure 4 As shown, in order to illustrate the function of the multi-sensor time delay estimation method provided in this embodiment, a specific example is provided below.

[0113] Acquire image data, perform feature detection on the image data to obtain feature points of the current image frame, obtain the corresponding feature points of the reference image frame based on the feature points of the current image frame, project the feature points of the current image frame and the feature points of the reference image frame onto the normalized plane, and calculate the planar feature pixel velocity of the corresponding points.

[0114] The projection relationship from the three-dimensional spatial point to the pixel coordinate system is converted into a correspondence between the planar feature pixel velocity of the feature point and the rate of change of the direction vector of the corresponding three-dimensional spatial point according to the chain rule; the planar feature pixel velocity of the feature point is converted into the rate of change of the direction vector according to the correspondence; and the parameterized coordinates of the three-dimensional spatial point are corrected according to the rate of change of the direction vector and the multi-sensor delay parameters.

[0115] The corrected 3D spatial point is back-projected onto its parameterized coordinates to obtain a corrected 3D spatial point, which is then projected onto a target image frame using the camera pose. The corrected 3D spatial point is processed through the inverse process of back-projection to obtain its parameterized coordinates in the target image frame. The parameterized coordinates of the 3D spatial point corresponding to the feature point in the target image frame and the rate of change of the direction vector in the parameterized coordinates are determined. The actual parameterized coordinates of the 3D spatial point are corrected based on the rate of change of the direction vector and the multi-sensor delay parameters. The reprojection residual of the feature point is calculated using the corrected actual parameterized coordinates and the corrected parameterized coordinates of the 3D spatial point in the target image frame.

[0116] Then, a cost function can be constructed based on the reprojection residuals corresponding to multiple feature points. The Jacobian matrix of the reprojection residuals relative to state variables such as multi-sensor delay parameters, IMU state, and pose state can be solved based on the cost function. The IMU integral constraints (integral constraints of the inertial measurement unit) can be determined based on the IMU data (acquired data of the inertial measurement unit) and timestamp offset. An optimization problem can be constructed by combining the IMU integral constraints and the Jacobian matrix of state variables such as multi-sensor delay parameters, IMU state, and pose state. The optimization problem can be solved using sparse matrix decomposition to obtain state variables such as multi-sensor delay parameters, IMU state, and pose state, and each state variable can be updated.

[0117] It should be noted that there are many ways to determine the IMU integration constraint based on IMU data and timestamp offset, such as integration after fixed delay compensation, resampling, etc., and this invention does not limit this method.

[0118] There are many ways to determine the timestamp offset based on the feature points obtained from feature extraction, and you can choose according to your actual needs. This embodiment does not limit this method.

[0119] The Jacobian matrix of the multi-sensor delay parameters indicates how the reprojection residual changes accordingly when the multi-sensor delay parameters change. It quantifies the degree and direction of the impact of adjusting the timestamp on the accuracy of state estimation, providing crucial optimization guidance on which direction and by how much to adjust the multi-sensor delay parameters to reduce errors. The meanings of the Jacobian matrices of other state variables are essentially the same as those of the Jacobian matrix of the multi-sensor delay parameters, and will not be elaborated upon here.

[0120] It should be noted that feature points of the current image frame can be determined from image data of the visual sensor through feature extraction. The feature points in the database are then traversed to find feature points with the same name for data association. After data association, the reprojection residuals are calculated for the feature points. Based on the reprojection residuals, the Jacobian matrices of each state variable are calculated. The reprojection residuals and the Jacobian matrices are then combined to construct constraint factors containing multi-sensor delay parameters. The goal is to minimize the sum of all constraint factors while simultaneously optimizing the multi-sensor delay parameters and other state variables to solve the optimization problem, and then update the multi-sensor delay parameters and other state variables.

[0121] Taking multi-sensor delay parameters as an example, the multi-sensor delay parameters can be assigned to the state of each frame, and the timestamp of the data collected by the visual sensor can be finely adjusted so that it reflects the physical state of the carrier at the same time as other sensors.

[0122] The multi-sensor time delay estimation method provided by this invention can convert the planar feature pixel velocity of the feature point into the rate of change of the direction vector of the corresponding three-dimensional spatial point according to the projection relationship, and explicitly incorporate the multi-sensor time delay parameters into the construction of the reprojection residual. In this way, the optimization problem constructed based on the reprojection residual and IMU integral constraints is solved, and the multi-sensor time delay parameters are optimized together. By updating the multi-sensor time delay parameters, the observation data from different sensors can be aligned in time, solving the systematic error of multimodal data state estimation and improving the accuracy of the state estimation results.

[0123] The multi-sensor time delay estimation device provided by the present invention is described below. The multi-sensor time delay estimation device described below can be referred to in correspondence with the multi-sensor time delay estimation method described above.

[0124] Figure 5 This is a schematic diagram of the multi-sensor time delay estimation device provided by the present invention, as shown below. Figure 5 As shown, the device includes the following modules.

[0125] The parameterized data determination module 510 is used to acquire image data from a vision sensor to determine the parameterized coordinates of a three-dimensional spatial point in space and the rate of change of the direction vector in the parameterized coordinates based on the image data.

[0126] The parameterized data correction module 520 is used to correct the parameterized coordinates of the three-dimensional spatial point based on the rate of change of the direction vector and the multi-sensor delay parameters.

[0127] The optimization problem construction module 530 calculates the reprojection residual of the corrected parameterized coordinates of the three-dimensional space points projected onto the target image frame to construct the optimization problem.

[0128] The time delay parameter solving module 540 is used to solve the optimization problem to obtain the multi-sensor time delay parameters.

[0129] Based on any of the above embodiments, the parameterized data determination module 510 is specifically used to determine the feature points of the current image frame according to the image data, and generate corresponding three-dimensional spatial points through the feature points; and to parameterize the three-dimensional spatial points according to the projection model to determine the parameterized coordinates of the three-dimensional spatial points in space.

[0130] Based on any of the above embodiments, the parameterized data determination module 510 is specifically used to determine the feature points of the current image frame according to the image data, and calculate the planar feature pixel velocity of the feature points; and convert the planar feature pixel velocity of the feature points into the rate of change of the direction vector of the three-dimensional spatial point corresponding to the feature points according to the projection relationship.

[0131] Based on any of the above embodiments, the parameterized data determination module 510 is specifically used to obtain the same feature points in the reference image frame based on the feature points of the current image frame; obtain the first coordinates of the same feature points in the reference image frame, the second coordinates of the feature points in the current image frame, and the time difference between the reference image frame and the current image frame, so as to calculate the planar feature pixel velocity of the feature points.

[0132] Based on any of the above embodiments, the parameterized data determination module 510 is specifically used to convert the projection relationship from the three-dimensional space point to the pixel coordinate system into a correspondence between the planar feature pixel velocity of the feature point and the rate of change of the direction vector of the three-dimensional space point corresponding to the feature point according to the chain rule; and to convert the planar feature pixel velocity of the feature point into the rate of change of the direction vector of the three-dimensional space point corresponding to the feature point according to the correspondence.

[0133] Based on any of the above embodiments, the optimization problem construction module 530 is specifically used to back-project the parameterized coordinates of the corrected three-dimensional spatial points to obtain the corrected three-dimensional spatial points, so as to project the corrected three-dimensional spatial points onto the target image frame using the camera pose; and to process the corrected three-dimensional spatial points through the inverse process of back-projection to obtain the parameterized coordinates of the corrected three-dimensional spatial points in the target image frame, so as to calculate the reprojection residual.

[0134] Based on any of the above embodiments, the optimization problem construction module 530 is specifically used to determine the parameterized coordinates of the three-dimensional spatial points corresponding to the feature points of the target image frame and the rate of change of the direction vector in the parameterized coordinates; correct the actual parameterized coordinates of the three-dimensional spatial points according to the rate of change of the direction vector and the multi-sensor delay parameters; and calculate the reprojection residual of the feature points using the corrected actual parameterized coordinates and the corrected parameterized coordinates of the three-dimensional spatial points in the target image frame.

[0135] Figure 6 An example is a schematic diagram of the structure of an XR device, such as... Figure 6 As shown, the XR device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640. The processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a multi-sensor time delay estimation method. This method includes: acquiring image data from a visual sensor to determine the parametric coordinates of a three-dimensional spatial point and the rate of change of the direction vector in the parametric coordinates based on the image data; correcting the parametric coordinates of the three-dimensional spatial point based on the rate of change of the direction vector and multi-sensor time delay parameters; calculating the reprojection residual of the corrected parametric coordinates of the three-dimensional spatial point projected onto a target image frame to construct an optimization problem; and solving the optimization problem to obtain the multi-sensor time delay parameters.

[0136] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0137] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multi-sensor time delay estimation method provided by the above methods. The method includes: acquiring image data from a vision sensor to determine the parametric coordinates of a three-dimensional spatial point and the rate of change of the direction vector in the parametric coordinates based on the image data; correcting the parametric coordinates of the three-dimensional spatial point based on the rate of change of the direction vector and the multi-sensor time delay parameters; calculating the reprojection residual of the corrected parametric coordinates of the three-dimensional spatial point projected onto a target image frame to construct an optimization problem; and solving the optimization problem to obtain the multi-sensor time delay parameters.

[0138] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the multi-sensor delay estimation method provided by the methods described above. This method includes: acquiring image data from a visual sensor to determine, based on the image data, the parameterized coordinates of a three-dimensional spatial point in space and the rate of change of the direction vector in the parameterized coordinates; correcting the parameterized coordinates of the three-dimensional spatial point based on the rate of change of the direction vector and multi-sensor delay parameters; calculating the reprojection residual of the corrected parameterized coordinates of the three-dimensional spatial point projected onto a target image frame to construct an optimization problem; and solving the optimization problem to obtain the multi-sensor delay parameters.

[0139] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0140] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-sensor time delay estimation method, characterized in that, include: Image data from a vision sensor is acquired to determine the parameterized coordinates of a three-dimensional spatial point and the rate of change of the direction vector in the parameterized coordinates based on the image data; wherein, the direction vector refers to the unit vector pointing from the camera center of the vision sensor to the three-dimensional spatial point; The parameterized coordinates of the three-dimensional spatial point are corrected based on the rate of change of the direction vector and the multi-sensor delay parameters. The reprojection residuals of the corrected parameterized coordinates of the three-dimensional spatial points onto the target image frame are calculated to construct an optimization problem; The multi-sensor delay parameters are obtained by solving the optimization problem; when the reprojection residual is minimized, the corresponding multi-sensor delay parameters are obtained. Determining the rate of change of the direction vector in the parameterized coordinates based on the image data includes: Based on the image data, the feature points of the current image frame are determined, and the planar feature pixel velocity of the feature points is calculated; Based on the projection relationship, the planar feature pixel velocity of the feature point is converted into the rate of change of the direction vector of the corresponding three-dimensional spatial point of the feature point; Based on the projection relationship, the planar feature pixel velocity of the feature point is converted into the rate of change of the direction vector of the corresponding three-dimensional spatial point, including: According to the chain rule, the projection relationship from a three-dimensional spatial point to a pixel coordinate system is converted into a correspondence between the planar feature pixel velocity of the feature point and the rate of change of the direction vector of the corresponding three-dimensional spatial point. Based on the correspondence, the planar feature pixel velocity of the feature point is converted into the rate of change of the direction vector of the corresponding three-dimensional spatial point of the feature point; The calculation of the planar feature pixel velocity of the feature point includes: Based on the feature points of the current image frame, obtain the same feature points in the reference image frame; The first coordinates of the same feature point in the reference image frame, the second coordinates of the feature point in the current image frame, and the time difference between the reference image frame and the current image frame are obtained to calculate the planar feature pixel velocity of the feature point.

2. The multi-sensor time delay estimation method according to claim 1, characterized in that, Determining the parametric coordinates of a three-dimensional spatial point based on the image data includes: The feature points of the current image frame are determined based on the image data, and corresponding three-dimensional spatial points are generated using the feature points. The three-dimensional spatial points are parameterized according to the projection model to determine the parameterized coordinates of the three-dimensional spatial points in space.

3. The multi-sensor time delay estimation method according to claim 1, characterized in that, Calculating the reprojection residual of the corrected parameterized coordinates of the three-dimensional spatial points projected onto the target image frame includes: The corrected three-dimensional spatial points are obtained by back-projecting the parameterized coordinates of the corrected three-dimensional spatial points, and then the corrected three-dimensional spatial points are projected onto the target image frame using the camera pose. The corrected 3D spatial points are processed by the inverse process of back projection to obtain the parameterized coordinates of the corrected 3D spatial points in the target image frame, so as to calculate the reprojection residual.

4. The multi-sensor time delay estimation method according to claim 3, characterized in that, Calculating the reprojection residuals includes: Determine the parameterized coordinates of the three-dimensional spatial points corresponding to the feature points of the target image frame, as well as the rate of change of the direction vector in the parameterized coordinates; The actual parameterized coordinates of the three-dimensional spatial point are corrected based on the rate of change of the direction vector and the multi-sensor delay parameters. The reprojection residual of the feature point is calculated using the corrected actual parameterized coordinates and the corrected parameterized coordinates of the three-dimensional spatial point in the target image frame.

5. A multi-sensor time delay estimation device, characterized in that, include: The parameterized data determination module is used to acquire image data from a vision sensor, and to determine the parameterized coordinates of a three-dimensional spatial point in space and the rate of change of the direction vector in the parameterized coordinates based on the image data; wherein, the direction vector refers to the unit vector pointing from the camera center of the vision sensor to the three-dimensional spatial point; The parameterized data correction module is used to correct the parameterized coordinates of the three-dimensional spatial points based on the rate of change of the direction vector and the multi-sensor delay parameters. The optimization problem construction module calculates the reprojection residual of the corrected parameterized coordinates of the three-dimensional space points projected onto the target image frame to construct the optimization problem; The time delay parameter solving module is used to solve the optimization problem to obtain the multi-sensor time delay parameters; when the reprojection residual is minimized, the corresponding multi-sensor time delay parameters are obtained. The parameterized data determination module is specifically used to determine the feature points of the current image frame based on the image data, and calculate the planar feature pixel velocity of the feature points; convert the planar feature pixel velocity of the feature points into the rate of change of the direction vector of the corresponding three-dimensional spatial point according to the projection relationship; wherein, according to the chain rule, the projection relationship from the three-dimensional spatial point to the pixel coordinate system is converted into a correspondence between the planar feature pixel velocity of the feature points and the rate of change of the direction vector of the corresponding three-dimensional spatial point; convert the planar feature pixel velocity of the feature points into the rate of change of the direction vector of the corresponding three-dimensional spatial point according to the correspondence; obtain the same feature points in the reference image frame based on the feature points of the current image frame; obtain the first coordinates of the same feature points in the reference image frame, the second coordinates of the feature points in the current image frame, and the time difference between the reference image frame and the current image frame, so as to calculate the planar feature pixel velocity of the feature points.

6. An XR device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the multi-sensor time delay estimation method as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multi-sensor time delay estimation method as described in any one of claims 1 to 4.

Images