Visual positioning method, computer storage medium and computer program product

Abstract

The present application provides a visual positioning method, computer storage medium, and computer program product, wherein the visual positioning method includes: acquiring image frames captured by a rolling shutter camera; acquiring feature points in the image frames for visual positioning; acquiring IMU relative pose data, wherein the IMU relative pose data is determined based on IMU frame data output by the IMU between the start and end of exposure of the image frame; determining image row IMU relative pose data according to the image row to which the feature points belong based on the IMU relative pose data; and performing visual positioning solution based on the image row IMU relative pose data, the position of the feature points in the image frame, an inter-image measurement item determined by pre-integration of the IMU frame data, and a state quantity to be estimated. Through the present application embodiment, effective and accurate visual positioning can be performed.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of positioning technology, and in particular to a visual positioning method, a computer storage medium, and a computer program product. Background Art

[0002] Visual positioning is a positioning technology that uses images acquired by visual sensors (such as cameras) and then performs calculations and processing on them to determine the position and pose of the device equipped with the visual sensor. Visual Inertial Odometry (VIO) is one of the technical solutions for achieving visual positioning. This solution uses data acquired by cameras and IMUs to estimate the pose of each image frame / IMU and simultaneously recover three-dimensional data of the surrounding environment.

[0003] An existing visual odometry solution uses a global shutter camera. However, global shutter cameras are relatively expensive. Cost considerations prevent general consumer devices such as mobile phones and small drones from equipping them with global shutter cameras. Therefore, these devices cannot adopt existing visual odometry technology, limiting the application scalability of visual odometry.

[0004] Another visual odometry solution uses a rolling shutter camera. Rolling shutter cameras are cheaper than global shutter cameras, but unlike global shutter cameras, which expose all pixels simultaneously, rolling shutter cameras expose image rows (pixel rows) one by one. The exposure duration of the image is called the readout time. However, when the rolling shutter camera is in motion, the device's pose will differ when different image rows are exposed for the same image. Therefore, although VIO solutions using rolling shutter cameras are less expensive, due to differences in exposure principles, ensuring accurate state estimation using VIO solutions using rolling shutter cameras remains a challenge for those skilled in the art. Summary of the Invention

[0005] In view of this, an embodiment of the present application provides a visual positioning solution to at least partially solve the above-mentioned problem.

[0006] According to a first aspect of an embodiment of the present application, a visual positioning method is provided, including: acquiring image frames captured by a rolling shutter camera; acquiring feature points of the image frames for visual positioning; acquiring IMU relative pose data, wherein the IMU relative pose data is determined based on IMU frame data output by the IMU between the start and end of exposure of the image frame; determining image row IMU relative pose data according to the image row to which the feature points belong based on the IMU relative pose data; performing visual positioning solution based on the image row IMU relative pose data, the position of the feature points in the image frame, the inter-image measurement items determined based on pre-integration of the IMU frame data, and the state quantity to be estimated.

[0007] According to a second aspect of an embodiment of the present application, a computer storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the visual positioning method as described in the first aspect is implemented.

[0008] According to a third aspect of an embodiment of the present application, a computer program product is provided, comprising computer instructions, wherein the computer instructions instruct a computing device to perform operations corresponding to the visual positioning method as described in the first aspect.

[0009] According to the solution provided in the embodiments of the present application, for scenarios where visual positioning is performed based on images captured by a rolling shutter camera, to avoid the problem of inaccurate state estimation caused by the rolling shutter camera exposing each image row at different times while moving, resulting in pixels in each image row being acquired at different poses, according to existing global camera positioning methods, feature points are first extracted from the image frames captured by the rolling shutter camera. Then, the image row IMU relative pose data corresponding to each image row is determined based on the relative pose data of the IMU and the image row to which each feature point belongs. Visual positioning is then solved based on the relative pose data of each image row, the position of the feature point in the image frame, the inter-image measurement items, and the state quantity to be estimated. Due to the high-frequency characteristics of the IMU, the pose solution is performed based on the high-frequency collected IMU frame data and the image frames used by the rolling shutter camera. The IMU relative pose corresponding to the different IMU frame data obtained in a timely manner can be used as the basis for determining the pose of each image row in the image frame, more effectively reflecting the pose of the rolling shutter camera when it is exposed by image row. Therefore, the solution of the embodiments of the present application can perform effective and accurate visual positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0011] Figure 1 This is a flowchart of the steps of a visual positioning method according to the first embodiment of the present application;

[0012] Figure 2 This is a flowchart of the steps of a visual positioning method according to the second embodiment of the present application;

[0013] Figure 3A This is a flowchart of the steps of a visual positioning method according to the third embodiment of the present application;

[0014] Figure 3B for Figure 3A A schematic diagram of various algorithms involved in the relative pose calculation process of an IMU frame in the illustrated embodiment;

[0015] Figure 3C for Figure 3A A schematic diagram of various algorithms involved in a reprojection error calculation process in the illustrated embodiment;

[0016] Figure 4 4 is a structural block diagram of a visual positioning device according to the fourth embodiment of the present application;

[0017] Figure 5A Schematic diagram of the overall processing process of a visual positioning system according to the fifth embodiment of the present application;

[0018] Figure 5B for Figure 5A A schematic diagram of partial data creation and updating in the illustrated embodiment;

[0019] Figure 6 This is a structural diagram of an electronic device according to Example 6 of the present application. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field should fall within the scope of protection of the embodiments of the present application.

[0021] The specific implementation of the embodiment of the present application is further explained below in conjunction with the accompanying drawings of the embodiment of the present application.

[0022] Example 1

[0023] Reference Figure 1 , shows a step flow chart of a visual positioning method according to Example 1 of the present application.

[0024] The visual positioning method of this embodiment includes the following steps:

[0025] Step S100: Acquire image frames captured by a rolling shutter camera.

[0026] Devices using a visual positioning system require an image acquisition device. In the embodiments of this application, the image acquisition device is a rolling shutter camera. As previously described, a rolling shutter camera exposes pixels row by row. During camera motion, pixels for each row are acquired at different device positions, ultimately forming a complete image frame.

[0027] Step S200: Acquire feature points of the image frame for visual positioning.

[0028] In visual positioning, feature points are specific locations in an image frame that remain stable despite camera motion, such as corners, edges, or regions with these characteristics. In practical applications, those skilled in the art may employ any appropriate method to extract feature points from image frames based on practical needs, and this disclosure is not intended to limit this approach.

[0029] Because a rolling shutter camera exposes image lines one by one, a particular image line may have one or more feature points, or may have no feature points. It should be noted that in the embodiments of this application, unless otherwise specified, "plurality," "multiple," and other quantities related to "multiple" all refer to two or more.

[0030] Step S300: Obtain IMU relative posture data.

[0031] The IMU relative posture data is determined based on the IMU frame data output by the IMU between the start and end of exposure of the image frame.

[0032] For a rolling shutter camera, when capturing an image frame, exposure begins at the first row and ends at the last row. All captured image rows form the image frame. This exposure duration can also be considered the rolling shutter camera's readout time. Generally speaking, the clock source for the same device hosting the rolling shutter camera and the IMU is the same. Based on this, the IMU frame data corresponding to the image frame exposure start and end time can be obtained.

[0033] Compared to image frame acquisition, IMU frame data is collected at a higher frequency. That is, multiple frames of IMU data are typically collected within the aforementioned duration of capturing a single image frame. Furthermore, the relative pose corresponding to each IMU frame data can be calculated based on the IMU frame data, i.e., the IMU relative pose data. For example, if four frames of IMU data are collected simultaneously during the acquisition duration of image frames 1 and 2, resulting in four IMU frames, the relative pose of these four IMU frames can be calculated.

[0034] Step S400: Based on the IMU relative pose data, determine the image row IMU relative pose data according to the image row to which the feature point belongs.

[0035] As mentioned above, an image row may or may not have feature points. For image rows that do have feature points, they may have one or multiple feature points. Because all feature points in an image row are acquired at a single camera pose, the IMU-relative pose data for all feature points in the same image row is the same. Based on this, the image row IMU-relative pose data for those image rows with feature points can be determined based on the image row to which the feature points belong.

[0036] Furthermore, because IMU frame data is collected at a higher frequency than image frame data, when determining the image row IMU relative pose data, the solution can be performed based on the correspondence between the IMU frame data and the image rows. For example, if image rows 1 and 2 are collected before the first IMU frame data is collected, and image row 3 is collected between the first and second IMU frame data, then the image row IMU relative pose data corresponding to image rows 1 and 2 can be determined by processing the relative pose data of the first IMU frame data (such as interpolation), while the image row IMU relative pose data corresponding to image row 3 can be determined by processing the relative pose data of the second IMU frame data.

[0037] Step S500: Perform visual positioning solution based on the IMU relative pose data of the image row, the position of the feature point in the image frame, the inter-image measurement items determined by pre-integration of the IMU frame data, and the state quantity to be estimated.

[0038] The solution of visual positioning can usually be achieved through optimization equations or constraint equations for state estimation. In the optimization equations or constraint equations, in addition to the state quantity to be estimated, corresponding observation quantities and error terms are usually required. Specifically, in the embodiment of the present application, the observation quantities and error terms can be reflected by the image row IMU relative pose data, the position of the feature point in the image frame, and the inter-image measurement terms. Therefore, in this step, the visual positioning solution can be performed through the above-mentioned multiple reflected observation quantities and error terms, the state quantity to be estimated, and the preset optimization equations or constraint equations.

[0039] This embodiment addresses the problem of inaccurate state estimation caused by existing global camera positioning methods when visual positioning is performed based on images captured by a rolling shutter camera. To avoid the problem of inaccurate state estimation caused by the rolling shutter camera exposing each image row at different times while moving, resulting in pixels in each image row being acquired at different poses, the method first extracts feature points from the image frames captured by the rolling shutter camera. Then, based on the relative pose data from the IMU and the image rows to which each feature point belongs, the IMU relative pose data corresponding to each image row is determined. Visual positioning is then performed based on the IMU relative pose data for each image row, the position of the feature point in the image frame, inter-image measurements, and the state quantity to be estimated. Due to the high-frequency characteristics of the IMU, pose calculation is performed based on the high-frequency IMU frame data and the image frames used by the rolling shutter camera. The IMU relative poses corresponding to the different IMU frame data obtained in a timely manner can be used as the basis for determining the pose of each image row in the image frame, more effectively reflecting the pose of the rolling shutter camera when exposing each image row. Therefore, the solution of this embodiment can achieve effective and accurate visual positioning.

[0040] Example 2

[0041] Reference Figure 2 , shows a step flow chart of a visual positioning method according to Example 2 of the present application.

[0042] The visual positioning method of this embodiment includes the following steps:

[0043] Step S102: Acquire image frames captured by a rolling shutter camera.

[0044] When using a rolling shutter camera to capture image frames, it exposes pixels row by row. The pixels of each image row are acquired at different positions of the device and eventually form a complete image frame.

[0045] Step S104: Acquire feature points in the image frame for visual positioning.

[0046] In visual positioning, feature points are specific locations in an image frame that remain stable despite camera motion, such as corners, edges, or regions within the image frame. In practical applications, those skilled in the art may employ any appropriate method to extract feature points from image frames based on actual needs.

[0047] Step S106: Determine the image row to which each feature point belongs, and perform row number deduplication on the image row to which each feature point belongs.

[0048] In the image frames exposed image row by image row, a certain image row may have one or more feature points, or may not have feature points. In an embodiment of the present application, those image rows with feature points are obtained, and the row number of the image row to which each feature point belongs is extracted.

[0049] Different feature points may be located in the same image row or in different rows. For example, if one image row has only one feature point, while another has five, then these five feature points correspond to the same row number. By determining the row number corresponding to the feature point, we can provide a basis for subsequent unified processing of feature points in the same image row.

[0050] Because for an image row, the pixels therein have the same exposure time and corresponding posture, therefore, if there are feature points in the image row, the feature points in the image row can be processed uniformly to increase data processing speed and improve data processing efficiency. To this end, in an embodiment of the present application, the row numbers corresponding to the feature points will be deduplicated to make the row numbers unique. For example, the row number corresponding to feature point A is 3, the row number corresponding to feature point B is 5, the row number corresponding to feature point C is 3, and the row number corresponding to feature point D is 6. After the row number deduplication process, row numbers 3, 5, and 6 will be obtained, where row number 3 corresponds to feature points A and C at the same time.

[0051] Step S108: Obtain IMU relative posture data.

[0052] The IMU relative posture data is determined based on the IMU frame data output by the IMU between the start and end of exposure of the image frame.

[0053] Compared to image frame acquisition, IMU frame data is collected at a higher frequency. That is, multiple frames of IMU data are typically collected within the duration of a single image frame. Furthermore, the relative pose corresponding to each IMU frame data can be calculated based on the IMU frame data, i.e., the IMU relative pose data. For example, if four frames of IMU data are collected simultaneously in the time interval between image frames 1 and 2, resulting in four IMU frames, the relative pose of these four IMU frames can be calculated.

[0054] In addition, in a feasible manner of this embodiment, before obtaining the IMU relative posture data, it can also include: collecting IMU frame data, and caching the IMU frame data into a first cache and a second cache respectively (wherein the first cache and the second cache can be referred to as a double cache); determining the IMU initial posture corresponding to each frame of IMU frame data based on the IMU frame data in the first cache; and performing pre-integration based on the IMU frame data in the second cache to obtain a pre-integration result for each frame of IMU frame data. In this case, obtaining the IMU relative posture data can include: determining the IMU relative posture data of the frame of IMU frame data relative to the acquisition moment of the corresponding image frame based on the IMU initial posture and pre-integration result corresponding to each frame of IMU frame data.

[0055] In the above method, the IMU frame data acquisition operation for determining the initial IMU pose and the IMU pre-integration operation use the same cache, which may cause IMU data conflicts. To prevent the IMU pre-integration operation from being affected, this method caches two copies of the IMU data upon arrival, preventing conflicts during the IMU pre-integration alignment.

[0056] In one feasible approach, pre-integration is performed based on the IMU frame data in the second cache to obtain the pre-integration result for each frame of IMU frame data. This can be achieved by: determining the Jacobian matrix of the IMU initial pose relative to the IMU bias; and pre-integrating the IMU frame data in the second cache based on the IMU initial pose and the Jacobian matrix to obtain the pre-integration result for each frame of IMU frame data. By pre-integrating the IMU frame data, inter-image measurement items between two image frames can be obtained to optimize the IMU initial pose and obtain a more accurate pose. At the same time, in order to save the computing power required for IMU pre-integration when the IMU bias changes, the Jacobian matrix of the relative pose relative to the IMU bias can be calculated when pre-integrating the IMU frame data. Based on this Jacobian matrix and the initial pose, an accurate relative pose can be obtained while using less computing power.

[0057] Step S110: Based on the IMU relative pose data, determine the image row IMU relative pose data according to the image row to which the feature points belong after row number deduplication.

[0058] In this embodiment, due to the row number deduplication process, when determining the image row IMU relative pose data based on the image row to which the feature points belong based on the IMU relative pose data, the image row IMU relative pose data corresponding to the deduplicated image row can be determined. As previously described, all feature points in each image row correspond to the same image row IMU relative pose data.

[0059] Furthermore, in one feasible approach, after determining the image row IMU relative pose data, a data structure for recording the correspondence between the row number and the image row IMU relative pose data can be generated based on the row number of each feature point and the image row IMU relative pose data corresponding to the image row to which each feature point belongs. This data structure facilitates the management of each feature point and its relative pose, and also greatly facilitates the subsequent use of the feature point and its corresponding relative pose data.

[0060] For example, a key-value data structure map can be constructed, where the key is the row number (specifically, it can be expressed as image row) and the value is the relative pose of the image row IMU (specifically, it can be expressed as pose). The data structure example formed is map<image row,pose> .

[0061] Step S112: Perform visual positioning solution based on the IMU relative pose data of the image row, the position of the feature point in the image frame, the inter-image measurement items determined by pre-integration of the IMU frame data, and the state quantity to be estimated.

[0062] After determining the image row IMU relative pose data, the position of the feature point in the image frame, the inter-image measurement items determined by pre-integration of the IMU frame data, and the state quantity to be estimated, the optimization equations or constraint equations used for visual positioning solution can be determined, and the visual positioning solution can be performed based on this.

[0063] The above process is exemplified below in the form of a specific scenario example.

[0064] In this example, a rolling shutter camera is installed on an autonomous vehicle, capturing a complete image frame every 33ms. For simplicity, the exposure time for each image row is assumed to be 3ms. (The 33ms and 3ms mentioned above are merely examples and should not be interpreted by those skilled in the art as representing the actual acquisition period or row exposure time.) Five IMU frames are simultaneously acquired during a 33ms*3 (three image frame acquisition period) period. First, feature points are extracted from image frame A and their corresponding row numbers are obtained. For ease of illustration, the number of extracted feature points in this example is reduced to 10, designated feature points 1-10. Furthermore, the row numbers corresponding to these 10 feature points are deduplicated, resulting in deduplicated image rows, assumed to be image rows 1, 3, 7, 11, 23, and 27. While processing image frame A, the relative poses corresponding to the five IMU frames are also calculated, yielding five IMU relative pose data points. Assume that the five IMU relative pose data are O, P, L, M, and N, respectively (this example is for simple illustration only; those skilled in the art should understand that pose includes data such as position, velocity, and attitude), and that image row 1 is acquired between the start exposure time of image frame A and the time interval of the first IMU frame data, image rows 3 and 7 are acquired between the time interval of the first IMU frame data and the second IMU frame data, image row 11 is acquired between the time interval of the second IMU frame data and the third IMU frame data, image row 23 is acquired between the time interval of the third IMU frame data and the fourth IMU frame data, and image row 27 is acquired between the time interval of the fourth IMU frame data and the fifth IMU frame data. Based on this, the relative poses corresponding to image rows 1, 3, 7, 11, 23, and 27, i.e., the image row IMU relative pose data, are determined, and are assumed to be O', P1', P2', L', M', and N', respectively. Then, based on the equations for state estimation, such as the equations for maximum a posteriori estimation based on the least squares method, O', P1', P2', L', M' and N', the positions of feature points 1-10 in the image frame (such as x, y coordinates), and the pre-integration results of the IMU frame data used when obtaining the IMU relative pose data, the state of the autonomous driving vehicle can be estimated to obtain a state estimation result, which is the visual positioning result, which can characterize the positioning information of the autonomous driving vehicle.

[0065] As can be seen, through this embodiment, for scenarios where visual positioning is performed based on images captured by a rolling shutter camera, to avoid the problem of inaccurate state estimation caused by the rolling shutter camera exposing each row at different times while moving, resulting in each row of pixels being acquired at different poses, according to existing global camera positioning methods, feature points are first extracted from the image frames captured by the rolling shutter camera. After deduplication is performed based on the row numbers of the feature points, image rows containing feature points and having unique row numbers are obtained. Then, the relative pose corresponding to each image row after deduplication is determined based on the relative pose of the IMU frame data, and visual positioning is performed based on the relative pose corresponding to each image row after deduplication. Due to the high-frequency characteristics of the IMU, the pose solution is performed on the IMU frame data collected at high frequency, and the relative pose corresponding to different IMU frame data can be timely obtained, which serves as the basis for determining the pose of each image row, more effectively reflecting the pose of the rolling shutter camera when exposing the image row. Furthermore, after deduplication of row numbers, the number of image rows can be much smaller than the number of image rows in the entire image frame. Therefore, the solution of this embodiment can not only perform visual positioning effectively and accurately, but also greatly reduce the amount of data used for positioning, thereby improving the efficiency of visual positioning.

[0066] Example 3

[0067] 3 , there is shown a flowchart of the steps of a visual positioning method according to the third embodiment of the present application.

[0068] The visual positioning method of this embodiment includes the following steps:

[0069] Step S202: Acquire image frames captured by a rolling shutter camera.

[0070] As mentioned earlier, a rolling shutter camera exposes image lines one by one, ultimately forming an image frame.

[0071] Step S204: Acquire feature points of the image frame for visual positioning, and obtain row numbers corresponding to the feature points.

[0072] The specific implementation of feature point extraction for an image frame can be implemented by those skilled in the art in an appropriate manner according to actual needs, and the embodiments of the present application do not limit this. After the feature points are determined, the row number of the image row where the feature points are located can be determined, thereby obtaining the row number corresponding to each feature point.

[0073] Step S206: Deduplication is performed on the obtained row numbers to obtain deduplicated image rows.

[0074] Different feature points may correspond to different row numbers or the same row number. Based on this, row number deduplication can be performed according to the row numbers corresponding to each feature point, so that the row numbers of the image rows with feature points are finally unique.

[0075] During visual odometry processing, to ensure optimal computational efficiency, the extracted feature points are relatively sparse, and the row numbers corresponding to the extracted or tracked feature points may be repeated, resulting in a large number of repeated operations. To this end, this embodiment removes duplicate row numbers. Specifically, the extracted feature points are sorted by row number and duplicate row numbers are removed to avoid repeated operations in subsequent processing and save computing power.

[0076] Step S208: Obtain IMU relative posture data.

[0077] The IMU relative posture data is determined based on the IMU frame data output by the IMU between the start and end of exposure of the image frame.

[0078] In one feasible approach, the IMU frame data of the exposure start time and exposure end time in the cache (the IMU frame data has the same time information as the image frame) can be obtained based on the exposure start time and exposure end time of the entire image frame (which can also be considered as the camera readout time) to achieve data alignment processing of the IMU frame data and thereby obtain multiple IMU frame data; perform pose estimation on each IMU frame data to obtain the IMU relative pose data corresponding to each IMU frame data. Through data alignment, the image frame acquisition timestamp t i to t i + r IMU frame data between, where t r Represents the readout time of a rolling shutter camera.

[0079] Furthermore, in this embodiment, before acquiring the IMU relative pose data, the following steps may be performed: acquiring IMU frame data and caching the IMU frame data into a dual buffer comprising a first buffer and a second buffer; determining the IMU initial pose corresponding to each frame of IMU frame data based on the IMU frame data in the first buffer; and performing pre-integration based on the IMU frame data in the second buffer to obtain a pre-integration result for each frame of IMU frame data. Acquiring the IMU relative pose data then includes: determining the IMU relative pose data of each frame of IMU frame data relative to the corresponding image frame at the time of acquisition based on the IMU initial pose and pre-integration result corresponding to each frame of IMU frame data. Because at this stage, IMU frame data acquisition and IMU pre-integration operations utilize the same buffer, this may cause IMU frame data conflicts. To avoid impacting the IMU pre-integration operation, in this method, two copies of the IMU frame data are cached upon arrival, preventing conflicts during alignment during the IMU pre-integration operation.

[0080] In one feasible approach, pre-integration is performed based on the IMU frame data in the second buffer to obtain the pre-integration results for each frame of IMU frame data. This can be achieved by: determining the Jacobian matrix of the IMU initial pose relative to the IMU bias; and pre-integrating the IMU frame data in the second buffer based on the IMU initial pose and the Jacobian matrix to obtain the pre-integration results for each frame of IMU frame data. By pre-integrating the IMU frame data, the measurement items between the two image frames involved in the pre-integration (i.e., inter-image measurement items) can be obtained to optimize the IMU initial pose and obtain a more accurate pose. At the same time, in order to save the computing power required for IMU pre-integration when the IMU bias changes, the Jacobian matrix of the relative pose with respect to the IMU bias can be calculated when pre-integrating the IMU frame data. Based on this Jacobian matrix and the IMU initial pose, an accurate relative pose can be obtained while using less computing power.

[0081] The algorithms involved in the relative pose calculation process of an IMU frame are as follows: Figure 3B As shown, Figure 3B This involves both the calculation of the IMU initial pose for each frame of IMU data and the Jacobian matrix used for pre-integration of the IMU frame data. This Jacobian matrix is ​​used to obtain the IMU relative pose data in combination with the IMU initial pose, and is also used for subsequent updates of the IMU relative pose data.

[0082] Specifically:

[0083] For a new image frame, we first need to calculate the relative pose of each frame of IMU frame data obtained from the cache and after data alignment relative to the image frame time (image acquisition time), that is, the IMU initial pose. In addition, for the calculation of the relative pose of the IMU, we also need to consider the bias state of the IMU at the image frame time, including the accelerometer bias at the image frame time. and the gyroscope bias at the image frame time Based on this, the calculation formula for the initial pose of the IMU frame data is as follows: Figure 3B As shown in:

[0084]

[0085]

[0086]

[0087] in, is the position of the current IMU frame data relative to the image frame time, is the speed of the current IMU frame data relative to the image frame time, The quaternion rotation of the current IMU frame data relative to the image frame time; is the position of the previous IMU frame data relative to the image frame time, is the speed of the previous IMU frame data relative to the image frame time, is the quaternion rotation of the previous IMU frame data relative to the image frame time, a ci is the accelerometer observation at the current IMU frame data moment, is the accelerometer bias at the image frame time, is the gyroscope bias at the image frame time, ω ci is the gyroscope observation at the current IMU frame data moment, δt is the time interval between two IMU frames, R(*) is the quaternion to rotation matrix operation, Represents quaternion multiplication. Thus, the position, velocity, and attitude of each frame of IMU data are calculated, namely: For a newly collected frame of IMU frame data, the position, velocity, and attitude are the initial IMU posture.

[0088] Since the IMU relative pose data also needs to rely on pre-integration results, the Jacobian matrix can be used to optimize it and reduce the amount of data calculation. Figure 3B It is also shown that when pre-integrating the IMU, the Jacobian matrix of the relative pose bias needs to be calculated. The calculation formula is as follows Figure 3B As shown in:

[0089]

[0090]

[0091]

[0092]

[0093]

[0094] in, Represents a rotation pair The partial derivative of is the gyroscope bias at the image frame time, Represents the right Jacobian matrix of SO(3), R(*) is the quaternion to rotation matrix operation, Represents the pose of the j-th IMU frame data relative to the k+1 moment; Indicates speed The partial derivative of Represents the pose of the kth IMU frame data relative to time i; Indicates speed The partial derivative of Indicates position pair The partial derivative of Indicates position pair The partial derivative of ,δt is the time interval between two IMU frames, j represents the number of IMU frames, a k represents the accelerometer observation of the kth IMU data frame, is the accelerometer bias at the image frame time.

[0095] Based on this Jacobian matrix, the IMU frame data can be pre-integrated. Based on the pre-integration result and the previous IMU initial pose, the optimized IMU relative pose can be obtained. In addition, the Jacobian matrix can also save the computing power required for IMU pre-integration when the bias changes.

[0096] Step S210: Based on the IMU relative pose data, determine the image row IMU relative pose data according to the image row to which the feature points belong after row number deduplication.

[0097] After calculating the IMU relative pose data corresponding to each IMU frame data, each image row can determine the relative pose of each image row after row number deduplication based on the IMU relative pose data of the corresponding IMU frame data, that is, the image row IMU relative pose data.

[0098] In one feasible approach, since each image row has a different row time (exposure time), the row time of the image row to which the feature point belongs can be determined. The row time indicates the exposure time of the image row (which can also be considered the time when the image row begins to be exposed). The IMU relative pose data is interpolated based on the row time of each image row to obtain the image row IMU relative pose data.

[0099] When row number deduplication is performed, the row times of different image rows after row number deduplication cannot be the same. Based on this, the row time of each image row after row number deduplication can be determined. The row time is used to indicate the exposure time of the image row (which can also be considered the time when the image row exposure begins). The IMU relative pose data is interpolated based on the row time of each image row to obtain the image row IMU relative pose data corresponding to each image row after row number deduplication.

[0100] Because the IMU frame data acquisition time interval may correspond to multiple image rows, and for an image row, its row time usually falls within a certain IMU frame data acquisition time interval, interpolation processing based on the relative pose of the IMU frame data (i.e., IMU relative pose data) can obtain a more accurate relative pose corresponding to the image row (i.e., image row IMU relative pose data). Similarly, if there are multiple image rows in an IMU acquisition time interval, the relative pose of each image row is determined based on its own row time and the relative pose of the IMU frame data, which is also more accurate.

[0101] For example, first, according to the image frame acquisition time t i , the row number n of the current image row (after removing duplicate row numbers), the total number of rows N of the image frame, and the readout time t of the image frame r , calculate the time of the current image row Then, interpolation processing is performed based on the IMU relative pose data of the corresponding IMU frame data to calculate the relative pose of the current image row (image row IMU relative pose data) in, Indicates relative posture, The interpolation process can be implemented by those skilled in the art using any appropriate algorithm (including linear interpolation and nonlinear interpolation) according to actual needs, and the present application does not limit this.

[0102] Furthermore, in one feasible approach, after determining the image row IMU relative pose data corresponding to each image row, a data structure for recording the correspondence between the row number and the relative pose data can be generated based on the row number of each image row and the image row IMU relative pose data corresponding to each image row. This data structure facilitates the management of each feature point and its relative pose, and also greatly facilitates the subsequent use of the feature point and its corresponding relative pose data.

[0103] For example, a key-value data structure map can be constructed, where the key is the row number (specifically, it can be expressed as image row) and the value is the relative pose of the image row IMU (specifically, it can be expressed as pose). The data structure example formed is map<image row,pose> .

[0104] Step S212: Perform visual positioning solution based on the IMU relative pose data of the image row, the position of the feature point in the image frame, the inter-image measurement items determined by pre-integration of the IMU frame data, and the state quantity to be estimated.

[0105] After determining the above parameters, the relative position corresponding to the entire image frame can be determined based on the preset optimization equation or constraint equation, thereby achieving visual positioning.

[0106] Generally speaking, optimization equations or constraint equations typically require information about reprojection error (which can be obtained based on the image row IMU relative pose data and the positions of feature points in the image), information about IMU pre-integration error (an inter-image measurement term determined by pre-integrating the IMU frame data), as well as marginalization prior information and the state quantity to be estimated. In this embodiment, the reprojection error can optionally be improved to better represent the reprojection error of a rolling shutter camera, thereby obtaining more accurate visual positioning results.

[0107] Specifically, the projected position of each feature point can be determined based on its position in the image frame and the image row IMU relative pose data of the image row to which each feature point belongs; the reprojection error of each feature point can be determined based on the observed position and projected position of each feature point; and visual positioning solution can be performed based on the reprojection error, inter-image measurement items, and marginalization prior information. Unlike the conventional method of determining reprojection error based on the relative pose corresponding to the entire image, this embodiment determines the reprojection error based on the relative pose of the image row to which each feature point belongs (image row IMU relative pose data). This can effectively avoid the error caused by the rolling shutter camera's image row-by-image exposure and obtain a more accurate reprojection error.

[0108] In one feasible approach, the reprojection error of this embodiment can be calculated using the following formula:

[0109]

[0110] in:

[0111]

[0112]

[0113] in represents the observation position of the feature point, Indicates the projection position of the feature point; Indicates the coordinates of the lth feature point (landmark point) observed in the jth frame image, Indicates the coordinates of the lth feature point (landmark point) observed in the i-th frame image, Respectively represent the position of the image row where the feature points of the two frames i and j are located relative to the image frame time (acquisition time), where the q element represents the posture and the p element represents the position. are the poses to be estimated for the two frames i and j, respectively, where the R element represents the pose and the p element represents the position. is the external parameter between the pre-calibrated camera and IMU, express The inverse of λ l represents the inverse depth state of the lth feature point, Represents the back projection function, which is used to back-project the pixel position in the image frame to the normalized plane of the camera according to the camera intrinsic parameters. R(*) represents the transformation formula between quaternion and rotation matrix. Indicates all states to be estimated, is the pose and motion state of the image frame to be estimated in the sliding time window at time k, It is a set of feature points that are observed in at least two frames within the time window.

[0114] In order to further illustrate the above reprojection error, a schematic diagram of the various algorithms involved in the reprojection error calculation process is shown as follows: Figure 3C When calculating the reprojection error, it is necessary to use the preset reprojection error calculation formula ( Figure 3C The calculation is performed to obtain each feature point The reprojection error of all feature points is obtained

[0115] After determining the reprojection error, the inter-image measurement term determined by the IMU pre-integration error, and the prior information of the marginalization, visual positioning can be performed based on the preset constraint equation or optimization equation for state estimation. In this embodiment, the form of the constraint equation used is as follows:

[0116]

[0117] in, Indicates all states to be estimated, is the pose and motion state of the image frame to be estimated in the sliding time window at time k, is the position state, is the speed state, is the posture state, b a ,b g is the bias state, λ l ,l∈[0,m] is the inverse depth state of the lth feature point, and m is the total number of feature points. m ,H m} is the marginalized prior information, is the residual of the IMU observation model, i.e. the pre-integration error, A dataset representing all IMU observations. is the residual of the visual reprojection model, i.e., the reprojection error, It is a set of feature points that are observed in at least two frames within the time window.

[0118] Based on the above equations, effective state estimation can be performed to obtain accurate visual positioning based on the rolling shutter camera.

[0119] In addition, in one feasible approach, in order to eliminate the impact of bias changes on the estimation of the relative pose of the IMU frame data, it is also possible to determine whether it is necessary to update the relative pose of the historical IMU frame data before the current IMU frame data based on the degree of IMU bias change in the current IMU frame data during the visual positioning solution process; if an update is required, the relative pose corresponding to the historical IMU frame data is updated. Specifically, the Jacobian matrix of the IMU initial pose relative to the IMU bias can be used to perform a first-order linear approximation update on the relative pose corresponding to the historical IMU frame data.

[0120] For example, for a certain IMU frame data, we can first determine whether it is necessary to update the relative posture corresponding to the IMU frame data before the IMU frame data (i.e., historical IMU frame data) based on the degree of bias change after optimization, such as calculating the acceleration bias before and after optimization. With gyroscope The second norm of the difference when or When the change is large, such as the acceleration bias change Greater than 1*10 -3 Or gyroscope bias changes Greater than 1*10 -5 , the Jacobian matrix calculated during the pre-integration of each frame of IMU frame data before the IMU frame data can be used to perform a first-order linear approximation update on the relative posture obtained by pre-integration of each frame of IMU frame data before the IMU frame data.

[0121] It should be noted that the aforementioned relative pose update process for historical IMU frame data can be performed in parallel with the relative pose calculation process for new IMU frame data. For example, while obtaining the IMU relative pose data for the current IMU frame data based on the initial IMU pose and pre-integration result corresponding to the current IMU frame data, a first-order linear relative pose update can be performed on the historical IMU frame data prior to the current IMU frame data.

[0122] As can be seen, through this embodiment, for scenarios where visual positioning is performed based on images captured by a rolling shutter camera, to avoid the problem of inaccurate state estimation caused by the rolling shutter camera exposing each image row at different times while moving, resulting in pixels in each image row being acquired at different poses, according to existing global camera positioning methods, feature points are first extracted from the image frames captured by the rolling shutter camera, and duplicate processing is performed based on the row numbers of the feature points to obtain image rows that contain feature points and have unique row numbers. Then, the relative pose corresponding to each image row is determined based on the relative pose of the IMU frame data, and visual positioning is performed based on the relative pose corresponding to each image row. Due to the high-frequency characteristics of the IMU, the pose solution is performed on the IMU frame data collected at high frequency, and the relative pose corresponding to different IMU frame data can be obtained in a timely manner. This can serve as the basis for determining the pose of each deduplicated image row, more effectively reflecting the pose of the rolling shutter camera when it is exposed by image row. Furthermore, after deduplication of row numbers, the number of image rows can be much smaller than the number of image rows of the entire image. Therefore, the solution of this embodiment can not only perform visual positioning effectively and accurately, but also greatly reduce the amount of data used for positioning, thereby improving the efficiency of visual positioning.

[0123] Example 4

[0124] Reference Figure 4 , shows a structural block diagram of a visual positioning device according to Example 4 of the present application.

[0125] The visual positioning device of this embodiment includes: a row number module 302, which is used to obtain image frames captured by a rolling shutter camera; a feature point module 304, which is used to obtain feature points of the image frames used for visual positioning; an IMU pose module 306, which is used to obtain IMU relative pose data, wherein the IMU relative pose data is determined based on the IMU frame data output by the IMU between the start and end of exposure of the image frame; an image row pose module 308, which is used to determine the image row IMU relative pose data according to the image row to which the feature point belongs based on the IMU relative pose data; a positioning module 310, which is used to perform visual positioning solution based on the image row IMU relative pose data, the position of the feature point in the image frame, the inter-image measurement items determined by pre-integration of the IMU frame data, and the state quantity to be estimated.

[0126] Optionally, the image row pose module 308 is used to determine the row time of the image row to which the feature point belongs, and the row time is used to indicate the exposure time of the image row; interpolate the IMU relative pose data according to the row time of each image row to obtain the image row IMU relative pose data

[0127] Optionally, the IMU posture module 306 is also used to collect IMU frame data before obtaining the IMU relative posture data, and cache the IMU frame data in a first cache and a second cache respectively; determine the IMU initial posture corresponding to each frame of IMU frame data based on the IMU frame data in the first cache; perform pre-integration based on the IMU frame data in the second cache to obtain a pre-integration result for each frame of IMU frame data; and further, determine the IMU relative posture data of the frame of IMU frame data relative to the acquisition moment of the corresponding image frame based on the IMU initial posture and pre-integration result corresponding to each frame of IMU frame data.

[0128] Optionally, when the IMU posture module 306 performs pre-integration based on the IMU frame data in the second cache to obtain the pre-integration result of each frame of IMU frame data: determines the Jacobian matrix of the IMU initial posture relative to the IMU bias; and performs pre-integration processing on the IMU frame data in the second cache according to the IMU initial posture and the Jacobian matrix to obtain the pre-integration result of each frame of IMU frame data.

[0129] Optionally, the visual positioning device of this embodiment also includes: an update module 312, which is used to determine whether it is necessary to update the relative pose of the historical IMU frame data before the current IMU frame data according to the degree of change of the IMU bias of the current IMU frame data during the process of performing the visual positioning solution; if updating is required, the relative pose corresponding to the historical IMU frame data is updated.

[0130] Optionally, the update module 312 uses the Jacobian matrix of the IMU initial pose relative to the IMU bias to perform a first-order linear approximation update on the relative pose corresponding to the historical IMU frame data.

[0131] Optionally, the positioning module 310 is used to determine the projection position of each feature point based on the position of each feature point in the image frame and the image row IMU relative pose data of the image row to which each feature point belongs; determine the reprojection error of each feature point based on the observation position and projection position of each feature point; and perform visual positioning solution based on the reprojection error, the inter-image measurement item, the marginalization prior information and the state quantity to be estimated.

[0132] Optionally, the image row pose module 308 is used to deduplicate the row numbers of the image rows to which each feature point belongs; based on the IMU relative pose data, the image row IMU relative pose data is determined according to the image rows to which the feature points after deduplication are located.

[0133] Optionally, the image row pose module 308 is also used to generate a data structure for recording the correspondence between the row number and the image row IMU relative pose data after determining the image row IMU relative pose data, based on the row number of the image row to which each feature point belongs and the image row IMU relative pose data corresponding to the image row to which each feature point belongs.

[0134] The visual positioning device of this embodiment is used to implement the corresponding visual positioning methods in the aforementioned multiple method embodiments and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here. In addition, the functional implementation of each module in the visual positioning device of this embodiment can refer to the description of the corresponding parts in the aforementioned method embodiments, which will not be described in detail here.

[0135] Example 5

[0136] This embodiment applies the aforementioned visual positioning method to a visual positioning system. The overall processing process of the visual positioning system is shown in the following figure. Figure 5A shown.

[0137] Figure 5A In the overall processing process shown in the figure, on the one hand, the rolling shutter camera collects image frames, and the system extracts and tracks features based on the collected image frames to obtain corresponding feature points and their change information; at the same time, the IMU collects IMU frame data to form IMU frame data, and the IMU frame data is double-buffered, with one copy stored in the first cache. Figure 5A The RSC (Rolling Shutter Compensation) IMU cache in the image is stored in the second cache. Figure 5A Pre-integrated IMU buffer in .

[0138] Then, data alignment and duplicate image row elimination operations are performed.

[0139] Specifically, data alignment can obtain the image timestamp t i to t i + r IMU frame data between t i Indicates the acquisition start time of the rolling shutter camera, t r Represents the readout time of the rolling shutter camera. Using the same buffer as the IMU pre-integration operation during this phase may cause IMU data conflicts. To minimize impact on the IMU pre-integration operation, two copies of the IMU data are cached upon arrival. This prevents conflicts during IMU pre-integration alignment.

[0140] The purpose of removing duplicate image rows is to reduce computing power. In the processing of the visual positioning system, in order to ensure the optimized computing efficiency, the extracted feature points are relatively sparse, and the row numbers corresponding to the extracted or tracked feature points may be repeated. If the mapping relationship between the feature point ID and the relative pose is established in the subsequent processing, the feature point ID will be used as the mapping relationship.<feature id,pose> , or establish a mapping relationship between image row number and relative pose map<image row,pose> , then a large number of repeated operations will be performed. Therefore, in this embodiment, repeated image rows are eliminated, that is, the rows of extracted feature points are sorted and repeated row numbers are eliminated, so that repeated operations can be avoided in subsequent processing and computing power can be saved.

[0141] Next, after data alignment and repeated image row elimination operations are performed, RSC creation and updating are performed.

[0142] The process is as Figure 5B As shown in the figure, for a new image frame, we first need to calculate the relative pose (IMU initial pose) of each frame of IMU frame data relative to the image frame time (image acquisition time) after data alignment. The calculation formula is as follows:

[0143]

[0144]

[0145]

[0146] in, is the position of the current IMU frame data relative to the image frame time, is the speed of the current IMU frame data relative to the image frame time, The quaternion rotation of the current IMU frame data relative to the image frame time; is the position of the previous IMU frame data relative to the image frame time, is the speed of the previous IMU frame data relative to the image frame time, is the quaternion rotation of the previous IMU frame data relative to the image frame time, a ci is the accelerometer observation at the current IMU frame data moment, is the accelerometer bias at the image frame time, is the gyroscope bias at the image frame time, ω ci is the gyroscope observation at the current IMU data frame, δt is the time interval between two IMU frames, R(*) is the quaternion to rotation matrix operation, Represents quaternion multiplication.

[0147] At the same time, in order to save the computing power required for IMU pre-integration when the bias changes, it is necessary to calculate the Jacobian matrix of the relative pose to bias when pre-integrating the IMU. The calculation formula is as follows:

[0148]

[0149]

[0150]

[0151]

[0152]

[0153] in, Represents a rotation pair The partial derivative of is the gyroscope bias at the image frame time, Represents the right Jacobian matrix of SO(3), R(*) is the quaternion to rotation matrix operation, Represents the pose of the j-th IMU frame data relative to the k+1 moment; Indicates speed The partial derivative of Represents the pose of the kth IMU data frame relative to time i; Indicates speed The partial derivative of Indicates position pair The partial derivative of Indicates position pair The partial derivative of ,δt is the time interval between two IMU frames, j represents the number of IMU frames, a k represents the accelerometer observation of the kth IMU frame data, is the accelerometer bias at the image frame time.

[0154] Furthermore, the IMU can be pre-integrated based on the Jacobian matrix, and the initial posture of the IMU can be optimized based on the pre-integration result to obtain the optimized IMU relative posture (i.e., IMU relative posture data).

[0155] After completing the above IMU relative pose calculation, the row set after repeated image rows are removed is traversed. First, according to the image acquisition time t i , the number of current image rows n, the total number of rows N, and the readout time t r Calculate the time of the current image row Then, interpolation processing is performed based on the relative pose calculated by IMU to calculate the relative pose of the current image row For a row set, a map data structure can be constructed, where the key is the row number and the value is the relative position. The map data structure can be represented as map<image row,pose> This map data structure can greatly facilitate the subsequent acquisition of feature point information and improve data acquisition and processing efficiency.

[0156] In addition, for the relative pose calculated by the IMU frame data corresponding to the old (collected) image frame, first determine whether the RSC needs to be updated based on the degree of change in the bias of the optimized IMU, that is, calculate the acceleration bias before and after optimization. With gyroscope The second norm of the difference when or When the change is large, (for example, the acceleration bias changes Greater than 1*10 -3 Or gyroscope bias changes Greater than 1*10 -5 ), the Jacobian matrix calculated above can be used to perform a first-order linear approximation update on the relative pose obtained by the IMU pre-integration of each frame, and then the relative pose in the map is updated according to the row set. The calculation formula is as follows:

[0157]

[0158]

[0159]

[0160] in, and are the changes in the gyroscope bias and accelerometer bias, respectively. R(*) represents the rotation matrix to quaternion conversion operation. The meanings of other parameters can be found in the previous formulas and are not repeated here.

[0161] After obtaining the IMU relative pose corresponding to the image row to which each feature point belongs, the optimization item can be constructed.

[0162] Specifically, the optimization term construction mainly includes three parts: IMU pre-integration error, reprojection error, and prior information provided by marginalization. In this embodiment, the reprojection error is mainly improved, and RSC compensation is added to the reprojection error to further improve the accuracy. The position of the 3D point (feature point) is represented by inverse depth, and the reprojection error formula is as follows:

[0163]

[0164] in:

[0165]

[0166]

[0167] in, represents the observation position of the feature point, Indicates the projection position of the feature point; Indicates the coordinates of the lth feature point (landmark point) observed in the jth frame image, Indicates the coordinates of the lth feature point (landmark point) observed in the i-th frame image, Respectively represent the position of the image row where the feature points of the two frames i and j are located relative to the image frame time (acquisition time), are the poses to be estimated for the two frames i and j respectively, is the external parameter between the pre-calibrated camera and IMU, express The inverse of λ l represents the inverse depth state of the lth feature point, Represents the back projection function, which is used to back-project the pixel position in the image to the normalized plane of the camera according to the camera intrinsic parameters. R(*) represents the transformation formula between quaternion and rotation matrix.

[0168] After determining the IMU pre-integration error, reprojection error, and prior information provided by marginalization, the overall state of the image can be estimated based on the pre-set constraint equation. In this example, the constraint equation used is as follows:

[0169]

[0170] in, Indicates all states to be estimated, is the pose and motion state of the image frame to be estimated in the sliding time window at time k, is the position state, is the speed state, is the posture state, b a ,b g is the bias state, n is the number of image frame moments in the time window, λ l ,l∈[0,m] is the inverse depth state of the lth feature point, and m is the total number of feature points. m ,H m} is the marginalized prior information, is the residual of the IMU observation model, i.e. the pre-integration error, A dataset representing all IMU observations. is the residual of the visual reprojection model, i.e., the reprojection error, It is a set of feature points that are observed in at least two frames within the time window.

[0171] Through the above constraint equations, the overall state estimation result corresponding to the image frame can be obtained, and visual positioning can be achieved based on the state estimation result.

[0172] Through this embodiment, the high-frequency characteristics of the IMU are used to approximately simulate the movement of the rolling shutter camera during the readout time, thereby improving the effect of camera model approximation; at the same time, repeated image row elimination, dynamic update of RSC and other operations are used to reduce the consumption of computing power, while improving the visual positioning accuracy and ensuring its computational efficiency.

[0173] Example 6

[0174] Reference Figure 6 , shows a structural diagram of an electronic device according to embodiment six of the present application. The specific embodiments of the present application do not limit the specific implementation of the electronic device.

[0175] like Figure 6 As shown, the electronic device may include: a processor (processor) 502 , a communication interface (Communications Interface) 504 , a memory (memory) 506 , and a communication bus 508 .

[0176] in:

[0177] The processor 502 , the communication interface 504 , and the memory 506 communicate with each other via a communication bus 508 .

[0178] The communication interface 504 is used to communicate with other electronic devices or servers.

[0179] The processor 502 is configured to execute the program 510 , and specifically to execute the relevant steps in the above-mentioned visual positioning method embodiment.

[0180] Specifically, the program 510 may include program codes, which include computer operation instructions.

[0181] Processor 502 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs, or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0182] The memory 506 is used to store the program 510. The memory 506 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0183] The program 510 can be specifically used to enable the processor 502 to execute the visual positioning method described in any one of the aforementioned embodiments 1 to 3.

[0184] The specific implementation of each step in program 510 can be found in the corresponding descriptions of the corresponding steps and units in the above-mentioned visual positioning method embodiment, and will not be repeated here. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding process descriptions in the above-mentioned method embodiment, and will not be repeated here.

[0185] The electronic device of this embodiment performs visual positioning based on images captured by a rolling shutter camera. To avoid the problem of inaccurate state estimation caused by the rolling shutter camera exposing each image row at different times while moving, resulting in pixels in each image row being acquired at different poses, according to existing global camera positioning methods, the electronic device first extracts feature points from the image frames captured by the rolling shutter camera. Then, based on the IMU relative pose data and the image rows to which each feature point belongs, the IMU relative pose data corresponding to each image row is determined. Visual positioning is then performed based on the IMU relative pose data for each image row, the position of the feature point in the image frame, inter-image measurements, and the state quantity to be estimated. Due to the high-frequency characteristics of the IMU, pose calculation is performed based on the high-frequency IMU frame data collected and the image frames used by the rolling shutter camera. The relative poses corresponding to the different IMU frame data obtained in a timely manner can be used as the basis for determining the pose of each image row in the image frame, more effectively reflecting the pose of the rolling shutter camera when exposing each image row. Therefore, the solution of this embodiment can perform effective and accurate visual positioning.

[0186] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0187] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or as computer code originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded via a network and to be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by the computer, processor or hardware, the visual positioning method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the visual positioning method shown here, the execution of the code converts the general-purpose computer into a dedicated computer for executing the visual positioning method shown here.

[0188] Those skilled in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of this application.

[0189] The above implementation methods are only used to illustrate the embodiments of the present application, and are not intended to limit the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present application, and the scope of patent protection of the embodiments of the present application should be defined by the claims.

Claims

1. A visual positioning method, wherein: include: Get the image frames captured by the rolling shutter camera; Acquire feature points of the image frame for visual positioning, wherein the feature points remain stable after the camera moves; Acquire IMU relative pose data, where the IMU relative pose data is determined based on IMU frame data output by the IMU between the start and end of exposure of the image frame; Based on the IMU relative pose data, determining the image row IMU relative pose data according to the image row to which the feature point belongs; Visual positioning solution is performed based on the IMU relative pose data of the image row, the position of the feature point in the image frame, the inter-image measurement items determined by pre-integration of the IMU frame data, and the state quantity to be estimated.

2. The method according to claim 1, wherein The determining, based on the IMU relative pose data and according to the image row to which the feature point belongs, the image row IMU relative pose data includes: Determine a row time of an image row to which the feature point belongs, where the row time is used to indicate an exposure time of the image row; The IMU relative pose data is interpolated according to the row time of each image row to obtain the image row IMU relative pose data.

3. The method according to claim 1, wherein Before obtaining the IMU relative posture data, the method further includes: Collecting IMU frame data, and buffering the IMU frame data in a first buffer and a second buffer respectively; Determine an IMU initial pose corresponding to each frame of IMU frame data based on the IMU frame data in the first buffer; Perform pre-integration based on the IMU frame data in the second buffer to obtain a pre-integration result for each frame of IMU frame data; The obtaining of IMU relative posture data includes: According to the IMU initial posture and pre-integration result corresponding to each frame of IMU frame data, the IMU relative posture data of the frame of IMU frame data relative to the acquisition moment of the corresponding image frame is determined.

4. The method according to claim 3, wherein: The pre-integration is performed based on the IMU frame data in the second buffer to obtain a pre-integration result of each frame of IMU frame data, including: Determine the Jacobian matrix of the IMU initial pose relative to the IMU bias; According to the IMU initial pose and the Jacobian matrix, the IMU frame data in the second cache is pre-integrated to obtain a pre-integration result of each frame of IMU frame data.

5. The method according to claim 3, wherein The method further comprises: During the visual positioning solution, determining whether it is necessary to update the relative pose of the historical IMU frame data before the current IMU frame data according to the degree of change of the IMU bias of the current IMU frame data; If updating is required, the relative pose corresponding to the historical IMU frame data is updated.

6. The method according to claim 5, wherein: Updating the relative pose corresponding to the historical IMU frame data includes: The Jacobian matrix of the IMU initial pose relative to the IMU bias is used to perform a first-order linear approximation update on the relative pose corresponding to the historical IMU frame data.

7. The method according to any one of claims 1 to 6, wherein: The visual positioning solution is performed based on the IMU relative pose data of the image row, the position of the feature point in the image frame, the inter-image measurement item determined by pre-integration of the IMU frame data, and the state quantity to be estimated, including: Determine the projection position of each feature point based on the position of each feature point in the image frame and the image row IMU relative pose data of the image row to which each feature point belongs; Determine the reprojection error of each feature point based on the observed position and projection position of each feature point; Visual positioning solution is performed according to the reprojection error, the inter-image measurement item, the marginalization prior information and the state quantity to be estimated.

8. The method according to claim 1, wherein The method further comprises: performing row number deduplication on the image rows to which each feature point belongs; The method of determining the image row IMU relative pose data based on the IMU relative pose data according to the image row to which the feature point belongs includes: determining the image row IMU relative pose data based on the IMU relative pose data according to the image row to which the feature point belongs after row number deduplication.

9. The method according to claim 8, wherein After determining the image row IMU relative pose data, the method further includes: According to the row number of the image row to which each feature point belongs and the image row IMU relative posture data corresponding to the image row to which each feature point belongs, a data structure for recording the correspondence between the row number and the image row IMU relative posture data is generated.

10. A computer storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the visual positioning method according to any one of claims 1 to 9 is implemented.

11. A computer program product, comprising computer instructions, wherein the computer instructions instruct a computing device to execute operations corresponding to the visual positioning method according to any one of claims 1 to 9.

Images