Monocular panoramic imaging method and device, electronic equipment and readable storage medium
By using extended Kalman filtering that integrates monocular images, IMU, and GPS data, the problem of image stitching misalignment in monocular panoramic imaging is solved, and accurate panoramic image updates are achieved.
Patent Information
- Application Number
- CN202510934212.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-07
- Publication Date
- 2025-10-31
AI Technical Summary
Existing monocular panoramic imaging methods are prone to misalignment during image stitching, leading to failure in global image updates.
By acquiring monocular images, IMU data, and GPS data, extended Kalman filtering is used to fuse visual motion pose, IMU data, and GPS data to perform self-motion state estimation. Based on this, historical panoramic images are stitched together with the current bird's-eye view.
Accurate positioning was achieved, stitching misalignment was reduced, and the accuracy of panoramic image updates was ensured.
Smart Images
Figure CN120881401A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of image technology, and in particular relates to a monocular panoramic imaging method, device, electronic device, and readable storage medium. Background Technology
[0002] Currently, the monocular panoramic imaging method involves acquiring the current image through a monocular sensor installed on the vehicle, converting the current image into a bird's-eye view, and then combining the panoramic image with the acquired bird's-eye view to update the panoramic image.
[0003] However, this method relies solely on image features, and in some scenarios, monocular images are prone to problems, leading to image stitching misalignment and failure of global image updates. Summary of the Invention
[0004] This application provides a monocular panoramic imaging method, apparatus, electronic device, and readable storage medium, which can solve the problem of global image update failure.
[0005] In a first aspect, embodiments of this application provide a monocular panoramic imaging method, including:
[0006] Acquire the current monocular image and the vehicle's IMU data and GPS data, wherein the monocular image includes environmental information surrounding the vehicle;
[0007] Convert the current monocular image into a bird's-eye view at the current moment;
[0008] Determine the visual motion posture at the current moment based on the bird's-eye view at the current moment and the panoramic view at the previous moment;
[0009] Based on extended Kalman filtering, the vehicle's current self-motion state estimate is determined according to the visual motion posture, IMU data, and GPS data at the current moment.
[0010] Based on the self-motion state estimation at the current moment, the historical panoramic image and the bird's-eye view at the current moment are stitched together to obtain a new panoramic image.
[0011] In one embodiment, determining the vehicle's current self-motion state estimate based on extended Kalman filtering, according to the current visual motion pose, IMU data, and GPS data, includes:
[0012] Based on the IMU data at the current moment and the state vector at the previous moment, the position estimate, velocity estimate and attitude estimate at the current moment are determined, and the IMU process noise covariance at the current moment is determined based on the Allan variance calibration results.
[0013] Based on the gyroscope zero-bias estimate and accelerometer zero-bias estimate of the previous moment, determine the gyroscope zero-bias estimate and accelerometer zero-bias estimate of the current moment.
[0014] Based on the visual motion posture at the current moment, the reprojection error is determined, and based on the inlier rate and feature distribution uniformity of the bird's-eye view at the current moment, the visual covariance is determined.
[0015] The pose estimate at the current moment is updated using the reprojection error to obtain the new pose at the current moment;
[0016] Using the reprojection error and the GPS data at the current moment, the position estimate at the current moment is updated to obtain the new position at the current moment;
[0017] The GPS covariance is determined based on the precision factor and signal-to-noise ratio of the GPS data at the current moment.
[0018] The self-motion state estimation at the current moment includes the new position, new attitude, and velocity estimation at the current moment, and the state vector at the previous moment includes the new position, new attitude, velocity estimation, gyroscope zero-bias estimation, and accelerometer zero-bias estimation at the previous moment.
[0019] In one embodiment, the method further includes:
[0020] If visual detection fails, the position estimate, velocity estimate and attitude estimate at the current moment are determined based on the IMU data at the current moment and the state vector at the previous first preset number of moments, and the IMU process noise covariance at the current moment is determined based on the Allan variance calibration result.
[0021] Based on the gyroscope zero-bias estimation and accelerometer zero-bias estimation of the previous first preset number of time moments, determine the gyroscope zero-bias estimation and accelerometer zero-bias estimation of the current time moment.
[0022] Based on the GPS data at the current moment, the position estimate at the current moment is updated to obtain the new position at the current moment;
[0023] The GPS covariance is determined based on the precision factor and signal-to-noise ratio of the GPS data at the current moment.
[0024] The self-motion state estimation at the current moment includes the new position, new attitude, and velocity estimation at the current moment, and the state vector at the previous first preset number of moments includes the new position, new attitude, velocity estimation, gyroscope zero bias estimation, and accelerometer zero bias estimation at the previous first preset number of moments.
[0025] In one embodiment, the step of stitching together the historical panoramic image and the current bird's-eye view based on the self-motion state estimation at the current moment to obtain a new panoramic image includes:
[0026] Based on the self-motion state estimate at the current moment, determine the transformation matrix;
[0027] Using the transformation matrix, the historical panoramic image is projected onto the coordinate system of the current bird's-eye view to obtain the transformed historical panoramic image;
[0028] The transformed historical panoramic image is weighted and fused with the current bird's-eye view to obtain the new panoramic image.
[0029] In one embodiment, the step of weightedly fusing the transformed historical panoramic image with the current bird's-eye view to obtain the new panoramic image includes:
[0030] For each pixel, the weight of the pixel in each image is determined according to the timestamp of the block to which the pixel belongs in each image, and the fused pixel value of the pixel is determined according to the pixel value and weight of the pixel in each image. The historical panoramic image includes panoramic images at a second preset number of times.
[0031] The new panoramic image is generated based on the fused pixel values of each pixel.
[0032] In one embodiment, determining the visual motion posture at the current moment based on the bird's-eye view at the current moment and the panoramic view at the previous moment includes:
[0033] The current bird's-eye view and the previous panoramic view are matched by feature points to obtain matching points;
[0034] Based on the matching points, calculate the homography matrix and fundamental matrix of the bird's-eye view at the current moment;
[0035] Calculate the confidence levels of the homography matrix and the fundamental matrix;
[0036] Based on the planarity detection results and the confidence levels of each matrix, the target matrix is determined from the homography matrix and the fundamental matrix.
[0037] Based on the target matrix, the visual motion posture at the current moment is determined.
[0038] In one embodiment, the method further includes:
[0039] Store the changed blocks and their corresponding timestamps in the new panoramic image, wherein the timestamp of the changed blocks is the timestamp of the bird's-eye view at the current moment.
[0040] Secondly, embodiments of this application provide a monocular panoramic imaging device, comprising:
[0041] The acquisition module is used to acquire the monocular image and the vehicle's IMU data and GPS data at the current moment. The monocular image includes environmental information around the vehicle.
[0042] An image processing module is used to convert the current monocular image into a bird's-eye view at the current moment;
[0043] The positioning module is used to determine the visual motion posture at the current moment based on the bird's-eye view at the current moment and the panoramic view at the previous moment.
[0044] It is also used to determine the vehicle's current self-motion state estimate based on extended Kalman filtering, according to the visual motion posture, IMU data, and GPS data at the current moment;
[0045] An imaging module is used to stitch together a historical panoramic image and a bird's-eye view image at the current moment based on the self-motion state estimation at the current moment to obtain a new panoramic image.
[0046] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method as described in any one of the first aspects above.
[0047] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method as described in any one of the first aspects above.
[0048] Fifthly, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to perform the method described in any one of the first aspects above.
[0049] The beneficial effects of the embodiments in this application compared with the prior art are:
[0050] This application embodiment provides local high-precision pose through visual motion posture, supplements high-frequency motion data with IMU data, and suppresses global drift with GPS data. By fusing the visual motion posture, IMU data, and GPS data at the current moment, the self-motion state estimate at the current moment is determined, achieving accurate positioning and obtaining an accurate self-motion state estimate. Based on the accurate self-motion state estimate, the historical panoramic image and the current bird's-eye view can be correctly stitched together to obtain a correctly stitched panoramic image, reducing the possibility of stitching misalignment and failure to update the panoramic image.
[0051] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description
[0052] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0053] Figure 1 This is a schematic flowchart of the first method for monocular panoramic imaging provided in an embodiment of this application;
[0054] Figure 2 This is a schematic diagram of the second process of a monocular panoramic imaging method provided in an embodiment of this application;
[0055] Figure 3 This is a schematic diagram of the structure of a monocular panoramic imaging device provided in an embodiment of this application;
[0056] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0057] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.
[0058] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.
[0059] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0060] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."
[0061] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0062] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.
[0063] In one embodiment, Figure 1 This is a schematic diagram of the first process of a monocular panoramic imaging method provided in an embodiment of this application. For example... Figure 1 As shown, the method includes:
[0064] S11: Acquire the current monocular image and vehicle IMU data and GPS data.
[0065] The monocular image includes information about the environment surrounding the vehicle.
[0066] In this application, a monocular sensor, an inertial sensor, and a GPS sensor are mounted on the vehicle. Multiple sets of data are collected using a checkerboard pattern, and the extrinsic parameter matrix is jointly optimized to calibrate the monocular sensor. Hardware triggering is used to align the timestamps of the monocular sensor, inertial sensor, and GPS sensor, with an error of less than 1ms.
[0067] For example, the monocular sensor is a single fisheye camera (field of view greater than 190°) or a wide-angle camera (field of view greater than 120°), with a resolution of 1920×1080fps. The inertial sensor is a 6-axis sensor (3-axis accelerometer + 3-axis gyroscope), with a sampling rate of 200Hz, zero-bias stability of ±0.01° / s (gyroscope), and ±0.02m / s.2 (Accelerometer). The GPS sensor has a positioning accuracy of ±1.5m and a timestamp alignment error of less than 1ms. The processor acquiring data from each sensor is an RK3588 embedded processor with a GPU, running Linux.
[0068] The system acquires a monocular image of the current moment using a monocular sensor, IMU data of the vehicle using an inertial sensor, and GPS data using a GPS sensor. The processor then obtains the monocular image, vehicle IMU data, and GPS data of the current moment.
[0069] S12: Convert the current monocular image into a bird's-eye view of the current moment.
[0070] In applications, the intrinsic parameters, pitch angle, and distortion coefficients of a monocular sensor are used to remove image distortion through a distortion model, and then the monocular image is converted into a bird's-eye view using a transformation formula.
[0071]
[0072] For example, the transformation formula is: (x,y,z) represents the three-dimensional coordinates of a pixel, fx and fy represent the focal lengths, and u0 and v0 represent the coordinates of the principal point.
[0073] S13: Determine the visual motion posture at the current moment based on the bird's-eye view at the current moment and the panoramic view at the previous moment.
[0074] In the application, feature points of the bird's-eye view at the current moment are extracted, and the visual motion posture at the current moment is calculated based on the feature points matched between the bird's-eye view at the current moment and the panoramic view at the previous moment.
[0075] S14: Based on extended Kalman filtering, determine the vehicle's current self-motion state estimate according to the current visual motion posture, IMU data, and GPS data.
[0076] In applications, the velocity, attitude, and position estimates for the current moment are determined based on the acceleration and angular velocity data from the IMU at the current moment.
[0077] The coordinate system of the GPS data was converted from WGS-84 latitude and longitude to the local ENU coordinate system to control alignment error. The alignment error was verified by fitting the GPS trajectory and the visual-inertial trajectory using the least squares method. When the verified alignment error was less than 0.5m, the GPS data was used.
[0078] Based on extended Kalman filtering, velocity estimation, attitude estimation, position estimation, visual motion attitude, and GPS data are fused to obtain the self-motion state estimate at the current moment.
[0079] During the positioning process, the extended Kalman filter dynamically adjusts the weights of data from each sensor. Visual data with high confidence levels receives a larger weight, while GPS data with long update cycles receives a smaller weight.
[0080] S15: Based on the self-motion state estimation at the current moment, stitch the historical panoramic view and the bird's-eye view at the current moment together to obtain a new panoramic view.
[0081] In the application, the historical panoramic view is aligned with the current bird's-eye view using the self-motion state estimation at the current moment. Then, the historical panoramic view and the current bird's-eye view are stitched together to obtain a new panoramic view.
[0082] This application embodiment provides local high-precision pose through visual motion posture, supplements high-frequency motion data with IMU data, and suppresses global drift with GPS data. By fusing the visual motion posture, IMU data, and GPS data at the current moment, the self-motion state estimate at the current moment is determined, achieving accurate positioning and obtaining an accurate self-motion state estimate. Based on the accurate self-motion state estimate, the historical panoramic image and the current bird's-eye view can be correctly stitched together to obtain a correctly stitched panoramic image, reducing the possibility of stitching misalignment and failure to update the panoramic image.
[0083] Understandably, inertial sensors continuously provide data, and IMU data accumulates errors, leading to a decrease in positioning accuracy. GPS data, on the other hand, provides periodic absolute position and velocity measurements, offering global correction. This reduces the accumulated errors in IMU data, corrects the IMU's navigation calculations, limits drift, improves positioning accuracy, and ensures the accuracy of self-motion state estimation.
[0084] In one embodiment, step S13 includes:
[0085] S131: Perform feature point matching between the current bird's-eye view and the previous panoramic view to obtain matching points.
[0086] In applications, the ORB algorithm can be used to extract the bird's-eye view at the current moment, and BFMatcher can be used to match the feature points of the bird's-eye view at the current moment with the panoramic view at the previous moment to obtain the matching points.
[0087] S132: Based on the matched points, calculate the homography matrix and fundamental matrix of the bird's-eye view at the current moment.
[0088] In the application, the RANSAC algorithm is used to filter feature points and obtain matching point pairs belonging to interior points. Based on the matching point pairs belonging to interior points, the homography matrix and fundamental matrix of the bird's-eye view at the current time are calculated.
[0089] The homography matrix and the fundamental matrix are used to describe the correspondence between two images. The homography matrix is suitable for planar scenes or purely rotated scenarios, while the fundamental matrix is suitable for more general scenarios, including those with parallax.
[0090] S134: Calculate the confidence scores of the homography matrix and the fundamental matrix.
[0091] In applications, the confidence levels of the homography matrix and the fundamental matrix are calculated based on the number of interior points, reprojection error, and planarity detection results.
[0092] Specifically, the number of interior points, reprojection error, and planarity detection results are normalized, and the weighted sum of each normalized result is calculated to obtain the confidence level of each matrix.
[0093] The confidence level of the matrix can be calculated by taking the number of interior points, reprojection error, and planarity detection results. This can improve the robustness of motion estimation and achieve better robustness in complex scenes (such as when there are planar regions like the ground and non-planar regions like buildings during movement).
[0094] S135: Based on the planarity detection results and the confidence levels of each matrix, determine the target matrix from the homography matrix and the fundamental matrix.
[0095] In application, if the confidence level of each matrix is greater than the preset confidence level, the matrix with the higher confidence level is selected as the target matrix.
[0096] If the confidence levels of the matrices are similar, the matrix with the better result is selected as the target matrix based on the planarity detection results.
[0097] If the confidence level of a certain matrix fails, select the valid matrix as the target matrix.
[0098] S136: Determine the visual motion pose at the current moment based on the target matrix.
[0099] In the application, the target matrix is decomposed to obtain the rotation matrix and translation vector. The visual motion pose at the current moment is determined based on the rotation matrix and translation vector.
[0100] This embodiment combines the planarity detection results and the confidence levels of each matrix to determine the target matrix from the homography matrix and the fundamental matrix, thereby identifying a more suitable matrix. Based on the more suitable matrix, the visual motion posture is determined more accurately, providing a basis for obtaining more accurate self-motion state estimation.
[0101] In one embodiment, when visual detection is valid, step S14 includes:
[0102] S141: Based on the IMU data at the current moment and the state vector at the previous moment, determine the position estimate, velocity estimate and attitude estimate at the current moment, and determine the IMU process noise covariance at the current moment based on the Allan variance calibration results.
[0103] The state vector of the previous moment includes the new position, new attitude, velocity estimate, gyroscope bias estimate, and accelerometer bias estimate.
[0104] The new position is the vehicle's position in the ENU coordinate system. The velocity estimate is the vehicle's velocity in the local coordinate system. The new attitude is the vehicle's rotation quaternion in the world coordinate system. The gyroscope bias estimate is the error of the IMU angular velocity measurement system. The accelerometer bias estimate is the error of the IMU accelerometer measurement system.
[0105] The initial position is initialized using GPS data; if GPS data is unavailable, it is initialized using visual motion attitude. In the initial attitude, pitch / roll is initialized using gravity alignment (accelerometer) in a static scene, and the yaw angle is initialized using GPS data or a magnetometer. All zero biases are initialized using preset calibration values and static estimates during the warm-up period.
[0106] In application, based on extended Kalman filtering, the position estimate, velocity estimate, and attitude estimate at the current moment are determined using the state transition equation and the IMU data at the current moment and the state vector at the previous moment.
[0107] The state transition equations include
[0108] IMU data is divided into blocks at time intervals, the average value of each time block is calculated, and the variance of these average values is further analyzed to obtain the Allan variance. The IMU process noise covariance at the current time is determined based on the Allan variance calibration results. If the Allan variance calibration results are invalid, the IMU process noise covariance is dynamically adjusted according to fixed parameters. These fixed parameters can be determined based on experience or a manual.
[0109] In this scenario, the IMU process noise covariance is pre-initialized. For example, the IMU process noise covariance is diag(0.01,0.01,0.01,0.1,0.1,0.1).
[0110] S142: Determine the gyroscope zero-bias estimate and accelerometer zero-bias estimate at the current moment based on the gyroscope zero-bias estimate and accelerometer zero-bias estimate at the previous moment.
[0111] In the application, based on the extended Kalman filter and incorporating the random error of the gyroscope, the gyroscope bias estimate for the current moment is determined according to the gyroscope bias estimate from the previous moment. Similarly, based on the random error of the accelerometer, the accelerometer bias estimate for the current moment is determined according to the accelerometer bias estimate from the previous moment.
[0112] S143: Determine the reprojection error based on the visual motion posture at the current moment, and determine the visual covariance based on the inlier rate and feature distribution uniformity of the bird's-eye view at the current moment.
[0113] In the application, the reprojection error is determined based on the visual observation model and the visual motion posture at the current moment. The visual motion posture includes both attitude and position.
[0114] Example, visual observation model: z vis =[p x p y p z q w q x q y q z ] T .
[0115] The inlier rate can be calculated using the RANSAC algorithm to select inliers. The uniformity of feature distribution is determined by analyzing the features of the current bird's-eye view. Based on the inlier rate and feature distribution uniformity of the current bird's-eye view, the visual covariance is determined. If the inlier rate is greater than a preset inlier rate, the visual covariance of the previous time step is increased by a factor of 3.
[0116] In this example, the visual covariance is pre-initialized in the actual scenario. The visual covariance is diag(0.1,0.1,0.1,0.05,0.05,0.05).
[0117] S144: Use the reprojection error to update the attitude estimate at the current time and obtain the new attitude at the current time.
[0118] In application, based on the extended Kalman filter, the attitude estimate at the current moment is updated using the reprojection error according to the attitude update equation, so as to obtain the new attitude at the current moment.
[0119] S145: Using the reprojection error and the GPS data at the current time, update the position estimate at the current time to obtain the new position at the current time.
[0120] In the application, based on the extended Kalman filter, the position estimate at the current time is updated using the reprojection error and the GPS data at the current time according to the position update equation, so as to obtain the new position at the current time.
[0121] S146: Determine the GPS covariance based on the precision factor and signal-to-noise ratio of the GPS data at the current moment.
[0122] In the application, the GPS covariance is determined based on the accuracy factor and signal-to-noise ratio. If the 2D plane coordinate accuracy factor is greater than a preset value, updates are disabled.
[0123] The current self-motion state estimation includes the current position, new attitude, and velocity estimation.
[0124] This embodiment corrects and updates the vehicle's state by using visual motion posture and GPS data, and calculates the covariance based on the characteristics of the data at the current moment to obtain an accurate covariance and thus an accurate self-motion state estimate.
[0125] In one embodiment, the method further includes:
[0126] S21: If visual detection fails, determine the position estimate, velocity estimate and attitude estimate at the current time based on the IMU data at the current time and the state vector at the previous first preset number of time steps, and determine the IMU process noise covariance at the current time based on the Allan variance calibration results.
[0127] The state vector for the first preset number of time steps includes the new position, new attitude, velocity estimate, gyroscope bias estimate, and accelerometer bias estimate for the first preset number of time steps.
[0128] In application, a monocular image failure can be determined if the number of matching point pairs in the current bird's-eye view is less than a preset number or the reprojection error is greater than a preset threshold. When visual failure occurs, the data is backed up historically, and the state vector is reverted to the state vector of the previous first preset number of time steps. By retaining the state vector of the previous first preset number of time steps, motion estimation of visual anomalies is used, providing a basis for historical data backtracking.
[0129] Based on the extended Kalman filter, the position estimate, velocity estimate, and attitude estimate at the current moment are determined by using the state transition equation and the state vector of the previous first preset number of moments, according to the IMU data at the current moment.
[0130] S22: Determine the gyroscope zero-bias estimate and accelerometer zero-bias estimate at the current moment based on the gyroscope zero-bias estimate and accelerometer zero-bias estimate at the previous first preset number of time moments.
[0131] S23: Update the current position estimate based on the current GPS data to obtain the new position at the current time.
[0132] S24: Determine the GPS covariance based on the precision factor and signal-to-noise ratio of the GPS data at the current moment.
[0133] In applications where visual detection fails, only GPS data is used for updates, which shortens the GPS data update cycle by up to 50%. Based on the extended Kalman filter, and according to the position update equation, the current position estimate is updated using the current GPS data to obtain the new position at the current moment.
[0134] The current self-motion state estimation includes the current position, new attitude, and velocity estimation.
[0135] In this embodiment, when visual detection fails, data is rolled back to the state vector of the previous first preset number of time steps to provide an accurate vehicle state for the current moment's self-motion estimation. Based on the state vector of the previous first preset number of time steps, IMU data and GPS data are fused to determine the current moment's self-motion state estimation, so as to obtain an accurate self-motion state estimation when visual detection fails, providing a foundation for accurate image stitching.
[0136] In one embodiment, step S15 includes:
[0137] S151: Determine the transformation matrix based on the self-motion state estimate at the current moment.
[0138] In applications, the transformation matrix is calculated based on the estimated position and pose of the vehicle at its current self-motion state.
[0139] S152: Using a transformation matrix, project the historical panoramic image onto the coordinate system of the current bird's-eye view to obtain the transformed historical panoramic image.
[0140] In applications, through perspective transformation or inverse perspective mapping, the historical panoramic image is projected onto the coordinate system (BEV) of the bird's-eye view using a transformation matrix, with the same spatial reference.
[0141] S153: The transformed historical panoramic image is weighted and fused with the current bird's-eye view to obtain a new panoramic image.
[0142] In one possible implementation, when the historical panorama includes panoramas of a second preset number of times, this can be achieved by maintaining a panorama queue of a second preset length. When a new frame arrives, the frame with the earliest timestamp is removed.
[0143] Correspondingly, step S153 includes:
[0144] S1531: For each pixel, determine the weight of the pixel in each image based on the timestamp of the block to which the pixel belongs in each image, and determine the fused pixel value of the pixel based on the pixel value and weight of the pixel in each image.
[0145] In this application, to better manage and stitch panoramic images, each panoramic image is divided into multiple blocks, each carrying a timestamp corresponding to the bird's-eye view at the time of the update. For each pixel, its position represents the block it belongs to. Using a preset weighting formula, the corresponding weight is determined based on the timestamp of the block to which the pixel belongs. The weight of each pixel is calculated for each image, obtaining the pixel's weight across all images. If the timestamp of the block to which it belongs is greater than a timeout threshold, the corresponding weight is directly set to zero.
[0146] Preset weight formula: λ is the attenuation factor, used to control the fusion ratio of the two images. current This is for setting up panoramic images based on the panoramic image queue. i Let be the timestamp of the i-th panoramic image in the panoramic image queue.
[0147] Using a preset weighting formula, the fused pixel value of a pixel is calculated based on the pixel value and weight of that pixel in each image.
[0148] Preset weighting formula: frame(x,y) represents the pixel value (grayscale value or color channel value) of the pixel at position (x,y). i is the weight, and n is the second preset quantity and the length of the panoramic image queue.
[0149] By performing time-weighted fusion of multi-frame panoramic images, short-term motion jitter can be suppressed and scene adaptability can be improved, so that the current bird's-eye view and historical panoramic images can be better stitched together, reducing stitching errors.
[0150] S1532: Generate a new panoramic image based on the fused pixel values of each pixel.
[0151] This embodiment determines the transformation matrix based on the self-motion state estimation at the current moment; using the transformation matrix, the historical panoramic image is projected onto the coordinate system of the bird's-eye view at the current moment to obtain the transformed historical panoramic image, so that the historical panoramic image can be better projected onto the coordinate system of the bird's-eye view at the current moment; and by weighted fusion of the transformed historical panoramic image and the bird's-eye view at the current moment, the bird's-eye view and the transformed historical panoramic image are better stitched together to obtain a new panoramic image with better stitching effect.
[0152] In one embodiment, Figure 2 This is a schematic diagram of the second process of a monocular panoramic imaging method provided in an embodiment of this application. For example... Figure 2 As shown, the method further includes:
[0153] S16: Store the changed blocks and their corresponding timestamps in the new panoramic image. The timestamp of the changed blocks is the timestamp of the current bird's-eye view.
[0154] In one possible implementation, the timestamp of the current bird's-eye view can be calculated based on the feature points matched between the current bird's-eye view and the panoramic view from the previous moment.
[0155] In the application, the changed portions between the new panoramic image and the previous panoramic image are identified, and the changed blocks are obtained. The changed blocks and timestamps are stored in the new panoramic image.
[0156] This embodiment reduces storage pressure by incrementally storing the changed blocks in the new panoramic image.
[0157] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. Furthermore, the data collection in the above embodiments is compliant, and its use or implementation does not involve any infringement upon public interests.
[0158] For ease of explanation, only the parts related to the embodiments of this application are shown in the description of the methods described in the above embodiments.
[0159] In one embodiment, Figure 3 This is a schematic diagram of the structure of a monocular panoramic imaging device provided in one embodiment of this application. Figure 3 As shown, the device includes:
[0160] The acquisition module 10 is used to acquire the current monocular image and the vehicle's IMU data and GPS data.
[0161] Image processing module 11 is used to convert the current monocular image into a bird's-eye view at the current moment.
[0162] The positioning module 12 is used to determine the visual motion posture at the current moment based on the bird's-eye view at the current moment and the panoramic view at the previous moment.
[0163] It is also used to determine the vehicle's current self-motion state estimate based on extended Kalman filtering, according to the current visual motion posture, IMU data, and GPS data.
[0164] The imaging module 13 is used to stitch together the historical panoramic image and the bird's-eye view at the current moment based on the self-motion state estimation at the current moment to obtain a new panoramic image.
[0165] In one embodiment, the positioning module is specifically configured to: determine the current position estimate, velocity estimate, and attitude estimate based on the current IMU data and the state vector from the previous time step; determine the IMU process noise covariance at the current time step based on the Allan variance calibration results; determine the gyroscope zero-bias estimate and accelerometer zero-bias estimate at the current time step based on the gyroscope zero-bias estimate and accelerometer zero-bias estimate from the previous time step; determine the reprojection error based on the current visual motion attitude; determine the visual covariance based on the inlier rate and feature distribution uniformity of the current bird's-eye view; update the attitude estimate at the current time step using the reprojection error to obtain the new attitude at the current time step; update the position estimate at the current time step using the reprojection error and the current GPS data to obtain the new position at the current time step; and determine the GPS covariance based on the accuracy factor and signal-to-noise ratio of the current GPS data.
[0166] The self-motion state estimation at the current moment includes the new position, new attitude, and velocity estimation at the current moment, while the state vector at the previous moment includes the new position, new attitude, velocity estimation, gyroscope bias estimation, and accelerometer bias estimation at the previous moment.
[0167] In one embodiment, the positioning module is further configured to, if visual detection fails, determine the position estimate, velocity estimate, and attitude estimate at the current moment based on the IMU data at the current moment and the state vector at the previous first preset number of moments, and determine the IMU process noise covariance at the current moment based on the Allan variance calibration result.
[0168] Based on the gyroscope zero-bias estimation and accelerometer zero-bias estimation of the previous first preset number of time moments, determine the gyroscope zero-bias estimation and accelerometer zero-bias estimation of the current time moment.
[0169] The current position estimate is updated based on the current GPS data to obtain the new position at the current time. The GPS covariance is determined based on the accuracy factor and signal-to-noise ratio of the current GPS data.
[0170] The self-motion state estimation at the current moment includes the new position, new attitude, and velocity estimation at the current moment, while the state vector at the previous first preset number of moments includes the new position, new attitude, velocity estimation, gyroscope bias estimation, and accelerometer bias estimation at the previous first preset number of moments.
[0171] In one embodiment, the imaging module is specifically used to determine a transformation matrix based on the self-motion state estimation at the current moment; using the transformation matrix, projecting the historical panoramic image onto the coordinate system of the bird's-eye view at the current moment to obtain the transformed historical panoramic image; and performing weighted fusion of the transformed historical panoramic image and the bird's-eye view at the current moment to obtain a new panoramic image.
[0172] In one embodiment, the device further includes a storage module.
[0173] The storage module is used to store the changed blocks and their corresponding timestamps in the new panoramic image. The timestamp of the changed block is the timestamp of the bird's-eye view at the current moment.
[0174] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 2 of this embodiment includes: at least one processor 20 ( Figure 4 (Only one is shown in the diagram), memory 21, and computer program 22 stored in said memory 21 and executable on said at least one processor 20, wherein said processor 20 executes said computer program 22 to implement the steps in any of the above method embodiments.
[0175] The electronic device 2 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. The electronic device 2 may include, but is not limited to, a processor 20 and a memory 21. Those skilled in the art will understand that... Figure 4 This is merely an example of electronic device 2 and does not constitute a limitation on electronic device 2. It may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, etc.
[0176] The processor 20 can be a Central Processing Unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.
[0177] In some embodiments, the memory 21 may be an internal storage unit of the electronic device 2, such as a hard disk or memory of the electronic device 2. In other embodiments, the memory 21 may be an external storage device of the electronic device 2, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 2. Furthermore, the memory 21 may include both internal and external storage units of the electronic device 2. The memory 21 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 21 can also be used to temporarily store data that has been output or will be output.
[0178] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.
[0179] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0180] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the above-described method embodiments.
[0181] This application provides a computer program product that, when run on an electronic device, enables the electronic device to implement the steps described in the various method embodiments above.
[0182] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some cases, the computer-readable medium cannot be an electrical carrier signal or a telecommunication signal.
[0183] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0184] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0185] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.
[0186] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0187] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. A monocular panoramic imaging method, characterized in that, include: Acquire the current monocular image and the vehicle's IMU data and GPS data, wherein the monocular image includes environmental information surrounding the vehicle; Convert the current monocular image into a bird's-eye view at the current moment; Determine the visual motion posture at the current moment based on the bird's-eye view at the current moment and the panoramic view at the previous moment; Based on extended Kalman filtering, the vehicle's current self-motion state estimate is determined according to the visual motion posture, IMU data, and GPS data at the current moment. Based on the self-motion state estimation at the current moment, the historical panoramic image and the bird's-eye view at the current moment are stitched together to obtain a new panoramic image.
2. The method according to claim 1, characterized in that, The step of determining the vehicle's current self-motion state estimate based on extended Kalman filtering, according to the current visual motion pose, IMU data, and GPS data, includes: Based on the IMU data at the current moment and the state vector at the previous moment, the position estimate, velocity estimate and attitude estimate at the current moment are determined, and the IMU process noise covariance at the current moment is determined based on the Allan variance calibration results. Based on the gyroscope zero-bias estimate and accelerometer zero-bias estimate of the previous moment, determine the gyroscope zero-bias estimate and accelerometer zero-bias estimate of the current moment. Based on the visual motion posture at the current moment, the reprojection error is determined, and based on the inlier rate and feature distribution uniformity of the bird's-eye view at the current moment, the visual covariance is determined. The pose estimate at the current moment is updated using the reprojection error to obtain the new pose at the current moment; Using the reprojection error and the GPS data at the current moment, the position estimate at the current moment is updated to obtain the new position at the current moment; The GPS covariance is determined based on the precision factor and signal-to-noise ratio of the GPS data at the current moment. The self-motion state estimation at the current moment includes the new position, new attitude, and velocity estimation at the current moment, and the state vector at the previous moment includes the new position, new attitude, velocity estimation, gyroscope zero-bias estimation, and accelerometer zero-bias estimation at the previous moment.
3. The method according to claim 2, characterized in that, Also includes: If visual detection fails, the position estimate, velocity estimate and attitude estimate at the current moment are determined based on the IMU data at the current moment and the state vector at the previous first preset number of moments, and the IMU process noise covariance at the current moment is determined based on the Allan variance calibration result. Based on the gyroscope zero-bias estimation and accelerometer zero-bias estimation of the previous first preset number of time moments, determine the gyroscope zero-bias estimation and accelerometer zero-bias estimation of the current time moment. Based on the GPS data at the current moment, the position estimate at the current moment is updated to obtain the new position at the current moment; The GPS covariance is determined based on the precision factor and signal-to-noise ratio of the GPS data at the current moment. The self-motion state estimation at the current moment includes the new position, new attitude, and velocity estimation at the current moment, and the state vector at the previous first preset number of moments includes the new position, new attitude, velocity estimation, gyroscope zero bias estimation, and accelerometer zero bias estimation at the previous first preset number of moments.
4. The method according to claim 1, characterized in that, The step of stitching together the historical panoramic image and the bird's-eye view at the current moment based on the self-motion state estimation at the current moment to obtain a new panoramic image includes: Based on the self-motion state estimate at the current moment, determine the transformation matrix; Using the transformation matrix, the historical panoramic image is projected onto the coordinate system of the current bird's-eye view to obtain the transformed historical panoramic image; The transformed historical panoramic image is weighted and fused with the current bird's-eye view to obtain the new panoramic image.
5. The method according to claim 4, characterized in that, The step of weightedly fusing the transformed historical panoramic image with the current bird's-eye view to obtain the new panoramic image includes: For each pixel, the weight of the pixel in each image is determined according to the timestamp of the block to which the pixel belongs in each image, and the fused pixel value of the pixel is determined according to the pixel value and weight of the pixel in each image. The historical panoramic image includes panoramic images at a second preset number of times. The new panoramic image is generated based on the fused pixel values of each pixel.
6. The method according to any one of claims 1 to 5, characterized in that, Determining the visual motion posture at the current moment based on the bird's-eye view at the current moment and the panoramic view at the previous moment includes: The current bird's-eye view and the previous panoramic view are matched by feature points to obtain matching points; Based on the matching points, calculate the homography matrix and fundamental matrix of the bird's-eye view at the current moment; Calculate the confidence levels of the homography matrix and the fundamental matrix; Based on the planarity detection results and the confidence levels of each matrix, the target matrix is determined from the homography matrix and the fundamental matrix. Based on the target matrix, the visual motion posture at the current moment is determined.
7. The method according to claim 6, characterized in that, Also includes: Store the changed blocks and their corresponding timestamps in the new panoramic image, wherein the timestamp of the changed blocks is the timestamp of the bird's-eye view at the current moment.
8. A monocular panoramic imaging device, characterized in that, include: The acquisition module is used to acquire the monocular image and the vehicle's IMU data and GPS data at the current moment. The monocular image includes environmental information around the vehicle. An image processing module is used to convert the current monocular image into a bird's-eye view at the current moment; The positioning module is used to determine the visual motion posture at the current moment based on the bird's-eye view at the current moment and the panoramic view at the previous moment. It is also used to determine the vehicle's current self-motion state estimate based on extended Kalman filtering, according to the visual motion posture, IMU data, and GPS data at the current moment; An imaging module is used to stitch together a historical panoramic image and a bird's-eye view image at the current moment based on the self-motion state estimation at the current moment to obtain a new panoramic image.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.