Panoramic imaging method, device, equipment and medium

Abstract

The invention discloses a panoramic imaging method and device, equipment and a medium, and relates to the technical field of automatic driving, and the method comprises the steps: obtaining current sensor data collected by a vehicle sensor, and carrying out the updating of a state vector of each particle in a current particle set through a particle filtering algorithm based on the current sensor data, and obtaining an updated state vector; calculating the observation likelihood corresponding to each particle based on the current sensor data and the updated state vector, and updating the weight of each particle by using the observation likelihood; determining an effective particle number based on the updated weight corresponding to each particle in the current particle set, and adjusting the particle number in the current particle set based on the effective particle number and scene complexity determined by using the statistical characteristics of the current sensor data to obtain an adjusted particle set; and determining optimal pose estimation according to the adjusted particle set, and fusing the historical panoramic image and the aerial view in the current sensor data by using the optimal pose estimation to generate a new panoramic image.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a panoramic imaging method, apparatus, device, and medium. Background Technology

[0002] Vehicle panoramic imaging technology is a core supporting technology for intelligent driving functions such as parking assistance and driving assistance. Monocular panoramic imaging solutions have become one of the mainstream choices for in-vehicle vision systems due to their low hardware cost and convenient installation. Traditional monocular panoramic imaging methods mainly rely on image feature matching to determine the camera pose, and then complete image stitching based on the pose estimation results. However, in practical applications, image features are easily affected by factors such as changes in lighting, missing textures, and dynamic objects, leading to feature matching failures or inaccurate pose estimation, which in turn causes image stitching misalignment and panoramic image update failure. This problem is particularly severe in weak texture scenes such as tunnels and underground parking garages.

[0003] Existing solutions typically employ Extended Kalman Filter (EKF) for multi-sensor fusion. While this approach improves positioning accuracy to some extent, EKF has inherent limitations: First, EKF assumes that system noise follows a Gaussian distribution, but in real-world vehicle scenarios, outliers in visual measurements and drift errors in sensor devices often exhibit non-Gaussian characteristics, leading to decreased filtering accuracy. Second, EKF requires a first-order Taylor expansion linearization approximation of the nonlinear motion model. In highly nonlinear motion scenarios such as sharp turns and rapid acceleration / deceleration, linearization errors accumulate, eventually causing filter divergence. Third, EKF can only maintain a single Gaussian pose assumption and cannot handle multimodal probability distribution scenarios such as multi-path selection at intersections, easily losing potentially correct poses and reducing system robustness.

[0004] In summary, improving the accuracy of pose estimation and the quality of image stitching in complex scenes is a problem that needs to be solved in the process of panoramic imaging. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a panoramic imaging method, apparatus, device, and medium that can improve the accuracy of pose estimation and image stitching quality in complex scenes during panoramic imaging. The specific solution is as follows: In a first aspect, this application discloses a panoramic imaging method, including: The system acquires current sensor data collected by vehicle sensors, and updates the state vector of each particle in the current particle set based on the current sensor data using a particle filtering algorithm to obtain the updated state vector; each particle carries a state vector and weights to represent the vehicle pose assumption. The observation likelihood for each particle is calculated based on the current sensor data and the updated state vector, and the weight of each particle is updated using the observation likelihood. The effective number of particles is determined based on the updated weight of each particle in the current particle set, and the number of particles in the current particle set is adjusted based on the effective number of particles and the scene complexity determined using the statistical characteristics of the current sensor data, so as to obtain the adjusted particle set. The optimal pose estimate is determined based on the adjusted particle set, and the optimal pose estimate is used to fuse the historical panoramic image and the bird's-eye view in the current sensor data to generate a new panoramic image.

[0006] Optionally, the current sensor data includes the current bird's-eye view generated after projecting and transforming the image captured by the vehicle's monocular camera, the current first motion data captured by the inertial measurement unit, and the current second motion data captured by the preset positioning system; the state vector includes the vehicle's position and velocity in the ENU coordinate system, the attitude quaternion used to describe the transition from the vehicle coordinate system to the ENU coordinate system, and the zero bias value of the inertial measurement unit. Accordingly, the step of updating the state vector of each particle in the current particle set based on the current sensor data using a particle filtering algorithm to obtain the updated state vector includes: Based on the current first motion data collected by the inertial measurement unit, the state vector of each particle in the current particle set is updated using a particle filtering algorithm and a preset kinematic model to obtain the updated state vector.

[0007] Optionally, the current first motion data includes the acceleration value measured by the accelerometer and the angular velocity value measured by the gyroscope; Accordingly, the process of updating the state vector of each particle in the current particle set based on the current first motion data collected by the inertial measurement unit and using a particle filtering algorithm and a preset kinematic model to obtain the updated state vector includes: The acceleration value is used to update the velocity and position in the state vector of each particle in the current particle set to obtain the updated velocity and updated position. The attitude quaternion in the state vector of each particle in the current particle set is updated based on the angular velocity value to obtain the updated attitude quaternion. The zero bias value in the state vector of each particle in the current particle set is updated based on the preset process noise to obtain the updated zero bias value. The updated state vector is obtained based on the updated velocity, the updated position, the updated attitude quaternion, and the updated zero bias.

[0008] Optionally, calculating the observation likelihood for each particle based on the current sensor data and the updated state vector includes: Obtain the feature matching results between the current bird's-eye view and the historical panoramic view, and calculate the visual observation likelihood corresponding to each particle based on the feature matching results and the updated state vector using the Huber loss function; Calculate the positioning observation likelihood for each particle based on the horizontal observation data in the current second motion data and the horizontal position in the updated state vector. Based on the preset ground plane height and the height position in the updated state vector, calculate the elevation observation likelihood for each particle; The joint observation likelihood corresponding to each particle is determined using the visual observation likelihood, the positioning observation likelihood, and the elevation observation likelihood. Accordingly, updating the weights of each particle using the observed likelihood includes: The weights of each particle are updated using the joint observation likelihood and then normalized to obtain the updated weights of each particle.

[0009] Optionally, the step of calculating the visual observation likelihood for each particle using the Huber loss function based on the feature matching result and the updated state vector includes: Based on the feature matching results, the matching feature points corresponding to each target feature point in the current bird's-eye view are determined in the historical panoramic view; Based on the updated state vector corresponding to each particle, the matching feature points in the historical panoramic image are reprojected onto the bird's-eye view coordinate system to obtain the corresponding reprojected feature points. Calculate the reprojection error between each target feature point and its corresponding reprojection feature point in the current bird's-eye view, and assign feature point weights to each target feature point based on the ORB response intensity of the target feature point and the distribution position of the target feature point in the current bird's-eye view; The reprojection error is weighted based on the feature point weights, and the Huber loss function is used to calculate the weighted reprojection error to obtain the error processing result. The visual observation likelihood for each particle is calculated based on the error processing result and the visual noise variance; wherein the visual noise variance is calculated based on the inlier rate corresponding to the feature matching result.

[0010] Optionally, the step of calculating the localization observation likelihood for each particle based on the horizontal observation data in the current second motion data and the horizontal position in the updated state vector includes: Extract the horizontal position from the updated state vector corresponding to each particle, and calculate the horizontal position error between the horizontal position and the horizontal observation data in the current second motion data; The position noise variance is determined based on the horizontal accuracy factor of the preset positioning system, and the positioning observation likelihood corresponding to each particle is calculated based on the horizontal position error and the position noise variance.

[0011] Optionally, the step of calculating the elevation observation likelihood for each particle based on the preset ground plane height and the elevation position in the updated state vector includes: Extract the height position from the updated state vector corresponding to each particle, and calculate the elevation error between the height position and the preset ground plane height; Based on the elevation error and the preset virtual observation variance, the elevation observation likelihood corresponding to each particle is calculated.

[0012] Optionally, adjusting the number of particles in the current particle set based on the effective particle count and the scene complexity determined using the statistical characteristics of the current sensor data to obtain an adjusted particle set includes: The scene complexity is determined based on the statistical characteristics of the current sensor data, and the target particle number is determined based on the effective particle number and the scene complexity. It is determined that the number of effective particles is less than a preset quantity threshold; If the number of effective particles is less than the preset number threshold, the current particle set is resampled based on the preset hierarchical resampling algorithm to obtain the sampled particle set, and the number of particles in the sampled particle set is adjusted based on the target number of particles to obtain the adjusted particle set. If the number of effective particles is not less than the preset quantity threshold, then the number of particles in the current particle set is adjusted based on the target number of particles to obtain the adjusted particle set. The number of particles in the sampled particle set is the same as the number of particles in the current particle set.

[0013] Optionally, determining the scene complexity based on the statistical characteristics of the current sensor data includes: The motion complexity is determined based on the standard deviation of acceleration and standard deviation of angular velocity of the inertial measurement unit within a preset continuous frame. The visual complexity is determined based on the total number of feature points extracted from the current bird's-eye view and the preset feature distribution uniformity; wherein, the preset feature distribution uniformity is determined based on the number of feature points in each grid in the current bird's-eye view and the average number of feature points in the grid. The complexity of the positioning system is determined based on the horizontal accuracy factor of the preset positioning system. The scene complexity is obtained by weighting the motion complexity, the visual complexity, and the positioning system complexity.

[0014] Optionally, the step of resampling the current particle set based on a preset hierarchical resampling algorithm to obtain a sampled particle set includes: The current particle set is arranged in descending order of weight to divide it into a preset high-weight layer, a preset medium-weight layer, and a preset low-weight layer. All particles in the preset high-weight layer are retained, and new particles are generated around the preset high-weight layer based on a preset noise strategy. Particles in the preset weighted layer are resampled according to a preset weight ratio; Discard all particles in the preset low-weight layer to obtain a sampled particle set based on all particles corresponding to the preset high-weight layer and all particles corresponding to the preset medium-weight layer.

[0015] Optionally, the process of adjusting the number of particles in the current particle set or the sampled particle set based on the target particle number includes: If the target number of particles is less than or greater than the number of particles, then the number of particles is adjusted to the target number of particles; If the number of target particles is equal to the number of particles, then the number of particles remains unchanged.

[0016] Optionally, the step of determining the optimal pose estimate based on the adjusted particle set, and fusing the optimal pose estimate with the historical panoramic image and the bird's-eye view from the current sensor data to generate a new panoramic image, includes: Based on the state vectors and weights of each particle in the adjusted particle set, a weighted average is calculated as the optimal pose estimate of the vehicle at the current moment. Based on the optimal pose estimation transformation matrix, the historical panoramic image is projected onto the bird's-eye view coordinate system using the transformation matrix, and the projected historical panoramic image is fused with the current bird's-eye view based on a preset time weighting coefficient to generate a new panoramic image.

[0017] Secondly, this application discloses a panoramic imaging device, comprising: The particle filtering module is used to acquire the current sensor data collected by the vehicle sensors, and based on the current sensor data, to update the state vector of each particle in the current particle set using a particle filtering algorithm to obtain the updated state vector; each particle carries a state vector and weights representing the vehicle pose assumption. The weight update module is used to calculate the observation likelihood of each particle based on the current sensor data and the updated state vector, and to update the weight of each particle using the observation likelihood. The particle number adjustment module is used to determine the effective number of particles based on the updated weight corresponding to each particle in the current particle set, and adjust the number of particles in the current particle set based on the effective number of particles and the scene complexity determined by the statistical characteristics of the current sensor data, so as to obtain the adjusted particle set. The panoramic image generation module is used to determine the optimal pose estimate based on the adjusted particle set, and to fuse the historical panoramic image and the bird's-eye view in the current sensor data using the optimal pose estimate to generate a new panoramic image.

[0018] Thirdly, this application discloses an electronic device, including: Memory, used to store computer programs; A processor is configured to execute the computer program to implement the steps of the aforementioned disclosed panoramic imaging method.

[0019] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the aforementioned disclosed panoramic imaging method.

[0020] As can be seen, this application acquires current sensor data collected by vehicle sensors, and updates the state vector of each particle in the current particle set using a particle filtering algorithm based on the current sensor data to obtain an updated state vector. Each particle carries a state vector and weights representing the vehicle pose hypothesis. The observation likelihood corresponding to each particle is calculated based on the current sensor data and the updated state vector, and the weights of each particle are updated using the observation likelihood. The effective number of particles is determined based on the updated weights of each particle in the current particle set, and the number of particles in the current particle set is adjusted based on the effective number of particles and the scene complexity determined using the statistical characteristics of the current sensor data to obtain an adjusted particle set. The optimal pose estimation is determined based on the adjusted particle set, and the optimal pose estimation is used to fuse the historical panoramic image and the bird's-eye view in the current sensor data to generate a new panoramic image.

[0021] Beneficial Effects: This application replaces the traditional EKF algorithm with a particle filter algorithm. It represents the posterior probability distribution of the vehicle pose using a set of particles carrying state vectors and weights, eliminating the need for the assumption of a noisy Gaussian distribution and linearization of the nonlinear motion model. For outliers in visual measurements, combined with observational likelihood calculation, their interference with particle weights can be effectively suppressed. Even in scenarios prone to non-Gaussian errors, such as weak textures and abrupt lighting changes, the accuracy of pose estimation remains intact, resolving the stitching misalignment problem caused by discrepancies between the noise distribution assumption and reality in traditional methods. This application determines scene complexity based on the statistical characteristics of current sensor data and dynamically adjusts the particle set size based on the effective particle count. This adaptive particle count adjustment strategy resolves the contradiction between resource waste in simple scenes and insufficient accuracy in complex scenes in traditional fixed particle count schemes. Finally, the optimal pose estimate output by the particle filter is used to fuse the historical panoramic image with the current bird's-eye view, ensuring the geometric consistency of the projection transformation. By incorporating constraints from multi-sensor observation data, the continuity of pose estimation can be maintained even in extreme scenarios such as visual failure. In this way, this application improves the accuracy of pose estimation and image stitching quality in complex scenes during panoramic imaging. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0023] Figure 1 This is a flowchart of a panoramic imaging method disclosed in this application; Figure 2 This is a schematic diagram of a panoramic imaging process disclosed in this application; Figure 3 This is a schematic diagram illustrating the calculation process of a joint observation likelihood disclosed in this application; Figure 4 This application discloses an adaptive particle number adjustment flowchart; Figure 5 This is a schematic diagram of a hierarchical resampling algorithm disclosed in this application; Figure 6 This is a schematic diagram of the structure of a panoramic imaging device disclosed in this application; Figure 7 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0025] In traditional monocular panoramic imaging, image features are easily affected by factors such as changes in illumination, missing textures, and dynamic objects, leading to feature matching failures or inaccurate pose estimation, which in turn causes image stitching misalignment and panoramic image update failure. Although Extended Kalman Filter (EKF) improves positioning accuracy to some extent, it has inherent limitations: First, EKF assumes that system noise follows a Gaussian distribution, but in actual vehicle scenarios, outliers in visual measurements and drift errors of sensor devices often exhibit non-Gaussian characteristics, resulting in decreased filtering accuracy. Second, EKF requires a first-order Taylor expansion linearization approximation of the nonlinear motion model. In highly nonlinear motion scenarios such as sharp turns and rapid acceleration / deceleration, linearization errors accumulate, eventually causing filter divergence. Third, EKF can only maintain a single Gaussian pose assumption and cannot handle multimodal probability distribution scenarios such as multi-path selection at intersections, easily losing potentially correct poses and reducing system robustness. Therefore, this application discloses a panoramic imaging method, apparatus, device, and medium, which can improve the accuracy of pose estimation and image stitching quality in complex scenes during panoramic imaging.

[0026] See Figure 1 and Figure 2 As shown in the figure, this application discloses a panoramic imaging method, which includes: Step S11: Obtain the current sensor data collected by the vehicle sensor, and based on the current sensor data, update the state vector of each particle in the current particle set using a particle filtering algorithm to obtain the updated state vector; each particle carries a state vector and weights used to represent the vehicle pose assumption.

[0027] In this embodiment, during vehicle operation, real-time sensor data collected by vehicle sensors is acquired. Based on this data, a particle filtering algorithm is used to update the state vector of each particle in the current particle set, resulting in an updated state vector. It is understood that the current particle set is the particle set output from the previous filtering process, and the total number of particles is within a preset range [100, 500]. Each particle in the set carries two core parameters: a state vector and a weight. The particle filtering maintains multiple particles, and the state vector of each particle represents a pose hypothesis of the vehicle. The state vector includes the vehicle's position and velocity in the ENU coordinate system, an attitude quaternion describing the transition from the vehicle coordinate system to the ENU coordinate system, and the zero bias value of the Inertial Measurement Unit (IMU). The IMU specifically includes a gyroscope and an accelerometer. It should be noted that the state vector is a 15-dimensional vector, which can be specifically represented as: ; in, The vehicle's position in the ENU (East-North-Up) coordinate system; The vehicle's velocity in the ENU coordinate system; The quaternion representation of the vehicle attitude (dimensionless, satisfying...) ), describing the rotation from the vehicle coordinate system to the ENU coordinate system; Zero bias for the gyroscope; The accelerometer has zero horizontal bias; x, y, and z correspond one-to-one with E, N, and U, respectively, where E, N, and U represent the east, north, and sky (up) directions, respectively.

[0028] The vertical (z-axis) zero bias of the accelerometer is estimated during system initialization using static gravity alignment and incorporated into the gravitational acceleration constant. It is no longer estimated separately during online filtering, simplifying the state dimension and improving computational efficiency. IMU measurements are performed in the vehicle coordinate system and updated in velocity and position after quaternion rotation transformation to the ENU coordinate system.

[0029] Furthermore, the weight of each particle is used to characterize the degree of fit between the pose hypothesis corresponding to that particle and the actual sensor observation data. The larger the weight value, the higher the credibility of the pose hypothesis. That is, this application uses a particle filtering algorithm to replace the traditional EKF algorithm, and uses a set of particles carrying state vectors and weights to represent the posterior probability distribution of the vehicle pose, without relying on the assumption of a noisy Gaussian distribution, and without linearizing the nonlinear motion model.

[0030] Step S12: Calculate the observation likelihood for each particle based on the current sensor data and the updated state vector, and update the weight of each particle using the observation likelihood.

[0031] In this embodiment, the key to particle filtering is calculating the observation likelihood of each particle, i.e., the probability that the observation conforms to the hypothesis of that particle, and then using the observation likelihood to update the particle weights. It should be noted that, for outliers in visual measurements, combining observation likelihood calculation can effectively suppress their interference with particle weights. Even in scenarios prone to non-Gaussian errors, such as weak textures and sudden changes in illumination, the accuracy of pose estimation can still be maintained, solving the splicing misalignment problem caused by the discrepancy between the noise distribution assumption and reality in traditional methods.

[0032] Step S13: Determine the effective number of particles based on the updated weights corresponding to each particle in the current particle set, and adjust the number of particles in the current particle set based on the effective number of particles and the scene complexity determined using the statistical characteristics of the current sensor data, so as to obtain the adjusted particle set.

[0033] In this embodiment, the effective number of particles is defined to assess the degree of particle degradation: ; Where N is the number of particles in the current particle set. Let i be the updated weight corresponding to the i-th particle. This reflects the degree of dispersion of particle weights. When all particle weights are equal, When the weight of a certain particle approaches 1, ; The smaller the value, the more severe the particle degradation.

[0034] Since the number of particles directly affects estimation accuracy and computational efficiency, too few particles cannot accurately represent complex distributions, while too many particles impose a heavy computational burden. Therefore, this application implements adaptive adjustment of the number of particles. Specifically, after updating the particle weights, this application first determines the effective number of particles based on the updated weights corresponding to each particle in the current particle set, and also determines the scene complexity using the statistical characteristics of the current sensor data. It should be noted that the scene complexity ranges from [0,1). A scene complexity close to 0 indicates a simple scene, corresponding to smooth motion, rich features, and good signal; a scene complexity close to 1 indicates a complex scene, corresponding to intense motion, sparse features, and poor GPS signal. Therefore, this application needs to adjust the number of particles in the current particle set based on the effective number of particles and the scene complexity to obtain the adjusted particle set.

[0035] In other words, this application determines the scene complexity based on the statistical characteristics of the current sensor data and dynamically adjusts the number of particles in the particle set in combination with the effective number of particles. This adaptive particle number adjustment strategy solves the contradiction between resource waste in simple scenes and insufficient accuracy in complex scenes in the traditional fixed particle number scheme.

[0036] Step S14: Determine the optimal pose estimate based on the adjusted particle set, and use the optimal pose estimate to fuse the historical panoramic image and the bird's-eye view in the current sensor data to generate a new panoramic image.

[0037] In this embodiment, the optimal pose estimate is determined based on the adjusted particle set, and then the optimal pose estimate output by the particle filter is used to fuse the historical panoramic image with the current bird's-eye view, thereby generating a new panoramic image. This method ensures the geometric consistency of the projection transformation, and by incorporating constraints from multi-sensor observation data, the continuity of pose estimation can be maintained even in extreme scenarios such as visual failure. Thus, this application improves the accuracy of pose estimation and the quality of image stitching in complex scenes during panoramic imaging.

[0038] In a specific implementation, the step of determining the optimal pose estimate based on the adjusted particle set and fusing the optimal pose estimate with the historical panoramic image and the bird's-eye view from the current sensor data to generate a new panoramic image includes: calculating a weighted average value as the optimal pose estimate of the vehicle at the current moment based on the state vector and weight of each particle in the adjusted particle set; projecting the historical panoramic image onto the bird's-eye view coordinate system using the transformation matrix based on the optimal pose estimate; and fusing the projected historical panoramic image with the current bird's-eye view based on a preset time weighting coefficient to generate a new panoramic image.

[0039] That is, in this embodiment, the weighted average value is first calculated based on the state vectors and weights of each particle in the adjusted particle set as the optimal pose estimate of the vehicle at the current moment: ; in, This represents the optimal pose estimate of the vehicle at the current moment; N0 represents the number of particles in the adjusted particle set; i represents the particle index, i.e., the i-th particle; Let be the weight of the i-th particle; Let be the state vector of the i-th example; the quaternion part needs to be calculated as a weighted average on the quaternion manifold.

[0040] Furthermore, based on the optimal pose Calculate the transformation matrix T from the historical panoramic view coordinate system to the current bird's-eye view coordinate system: ; Where R is the rotation matrix (obtained by quaternion conversion) and t is the translation vector.

[0041] The historical panoramic image is projected onto the current bird's-eye view coordinate system using a transformation matrix T. Then, based on a preset time weighting coefficient, the projected historical panoramic image and the current bird's-eye view are time-weightedly fused to generate a new panoramic image. Specifically: ; in, The pixel value at coordinates (x, y) for the new panoramic image generated by fusion; K is the length of the historical frame buffer queue participating in the fusion (e.g., K=5); k is the frame index in the buffer queue. The time weighting coefficient for the k-th frame; This represents the pixel value at (x, y) of the k-th frame of the historical image after it has been projected onto the current bird's-eye view coordinate system.

[0042] The time weighting coefficient is: ; in, As the attenuation factor, The timestamp of the k-th frame. At the current moment, K=5 is the length of the historical frame buffer queue.

[0043] As can be seen, this application uses a particle filter algorithm instead of the traditional EKF algorithm. It represents the posterior probability distribution of the vehicle pose using a set of particles carrying state vectors and weights, without relying on the assumption of a noisy Gaussian distribution or linearizing the nonlinear motion model. For outliers in visual measurements, the observation likelihood calculation effectively suppresses their interference with particle weights. Even in scenarios prone to non-Gaussian errors, such as weak textures or abrupt lighting changes, the accuracy of pose estimation is maintained, solving the stitching misalignment problem caused by discrepancies between the noise distribution assumption and reality in traditional methods. This application determines scene complexity based on the statistical characteristics of current sensor data and dynamically adjusts the particle set size based on the effective particle count. This adaptive particle count adjustment strategy resolves the contradiction between resource waste in simple scenes and insufficient accuracy in complex scenes in traditional fixed particle count schemes. Finally, the optimal pose estimate output by the particle filter is used to fuse the historical panoramic image and the current bird's-eye view, ensuring the geometric consistency of the projection transformation. By incorporating constraints from multi-sensor observation data, the continuity of pose estimation is maintained even in extreme scenarios such as visual failure. In this way, this application improves the accuracy of pose estimation and image stitching quality in complex scenes during panoramic imaging.

[0044] Building upon the aforementioned embodiments, it should be further noted that the vehicle sensor is a multi-source heterogeneous sensor combination, specifically including a monocular camera, an inertial measurement unit (IMU), a positioning system (such as GPS), and other sensors. These sensors work collaboratively to collect vehicle motion and environmental visual data at the current moment. Therefore, as... Figure 2 As shown, the current sensor data specifically includes the current bird's-eye view generated by projecting and transforming the image (e.g., 1920×1080@30fps) captured by the vehicle's monocular camera, the current first motion data captured by the inertial measurement unit (IMU), and the current second motion data captured by the preset positioning system (GPS). Furthermore, this application also requires timestamp alignment processing for the data collected by each sensor.

[0045] Specifically, the first motion data may include the acceleration value measured by the accelerometer and the angular velocity value measured by the gyroscope in the vehicle coordinate system, while the second motion data may include the position, velocity, and HDOP (Horizontal Dilution of Precision) in the ENU coordinate system.

[0046] In a specific implementation, the step of updating the state vector of each particle in the current particle set based on the current sensor data using a particle filtering algorithm to obtain an updated state vector includes: updating the state vector of each particle in the current particle set based on the current first motion data collected by the inertial measurement unit using a particle filtering algorithm and a preset kinematic model to obtain an updated state vector. That is, updating the state vector of each particle in the current particle set using a particle filtering algorithm is essentially a state prediction process based on IMU motion data. Furthermore, it should be noted that particle initialization is performed upon system startup. Position is initialized by GPS, pitch / roll is estimated using accelerometer gravity alignment in a stationary state, and the heading angle is initialized by GPS motion direction or magnetometer. An initial particle set is generated by sampling around the initial value according to a Gaussian distribution. ; in, For initial state estimation, For the initial covariance matrix, the diagonal elements are set as: location variance. speed variance Posture variance Zero bias During the motion model prediction phase, for each set of IMU data received (time interval Δt = 5 ms), motion prediction is performed for each particle. Let the state of the i-th particle at time k be... Predict the state at time k+1 .

[0047] Specifically, the current first motion data includes the acceleration value measured by the accelerometer and the angular velocity value measured by the gyroscope. Correspondingly, the process of updating the state vector of each particle in the current particle set based on the current first motion data collected by the inertial measurement unit, using a particle filtering algorithm and a preset kinematic model, to obtain an updated state vector includes: updating the velocity and position in the state vector of each particle in the current particle set using the acceleration value to obtain updated velocity and updated position; updating the attitude quaternion in the state vector of each particle in the current particle set based on the angular velocity value to obtain updated attitude quaternion; updating the zero bias value in the state vector of each particle in the current particle set based on preset process noise to obtain updated zero bias value; and obtaining the updated state vector based on the updated velocity, the updated position, the updated attitude quaternion, and the updated zero bias value.

[0048] Understandably, this embodiment first transforms the acceleration value from the vehicle coordinate system to the ENU coordinate system. Let the acceleration measurement in the vehicle coordinate system be... Through quaternions Corresponding rotation matrix Transform to ENU coordinate system: ; in, The acceleration vector in the vehicle coordinate system. The acceleration vector in the ENU coordinate system. Let be the rotation matrix of the i-th particle at time k, given by the corresponding quaternion. Calculations show that This is the gravity vector in the ENU coordinate system.

[0049] Velocity prediction update (ENU coordinate system): ; in, Let be the velocity of the i-th particle at time k. Let be the velocity of the i-th particle at time k+1. The IMU sampling interval (seconds, fixed value 0.005s, corresponding to a 200Hz sampling rate). This is the noise term for the speed process.

[0050] Furthermore, Expand into component form: ; ; ; in, , , These represent the eastward, northward, and skyward velocities of the i-th particle at time k+1; , , These represent the eastward, northward, and skyward velocities of the i-th particle at time k; , , These are the acceleration components in the east, north, and sky directions, respectively.

[0051] Location prediction update (ENU coordinate system): ; in, Let be the position of the i-th particle at time k+1. Let i be the position of the i-th particle at time k. Let be the velocity of the i-th particle at time k. This is noise from the position process.

[0052] Furthermore, Expand into component form: ; ; ; in, , , These represent the east, north, and sky positions of the i-th particle at time k+1; , , These represent the east, north, and sky positions of the i-th particle at time k.

[0053] Attitude prediction update (quaternion differential equation): ; in, Let k+1 be the temporary attitude quaternion after integration. Let be the attitude quaternion at time k. This is quaternion multiplication. This represents the angular velocity value measured by the gyroscope in the vehicle coordinate system. This is for attitude process noise. , , Let be the zero bias of the gyroscope for the i-th particle in the east, north, and sky directions.

[0054] To ensure the quaternion unit norm constraint, the prediction results are standardized: ; For the normalized quaternion, a first-order Euler approximation is used here, which is accurate enough for a small time step of 5ms. The normalization step eliminates the norm drift introduced by numerical integration, ensuring... .

[0055] Zero-biased prediction (random walk model): ; ; in, and The gyroscope and accelerometer have zero bias at time k+1. and The gyroscope and accelerometer have zero bias at time k. This refers to the noise during the gyroscope's zero-bias process. This refers to the noise during the zero-bias process of the accelerometer.

[0056] It should be noted that the above All data are process noise, following a zero-mean Gaussian distribution, and their covariance is determined by Allan variance calibration. The process noise covariance matrix is: ; The matrix elements correspond to the 15 components of the state vector: position (3), velocity (3), attitude (4), gyroscope bias (3), and accelerometer bias (2). In this way, each particle evolves independently according to its own state and the common IMU measurement, forming multiple hypothetical predictions of the future state.

[0057] As can be seen, this application constructs a multi-sensor joint observation model, unifying visual reprojection error, IMU prediction error, and GPS position error into a particle weight update framework. Particle filtering has strong capabilities in handling non-Gaussian distributions, representing probability distributions of arbitrary shapes without being limited by the Gaussian assumption. Particle filtering does not require linearization approximation, directly processing nonlinear motion models. In sharp turning scenarios, compared to the first-order Taylor expansion of EKF, particle filtering has smaller pose estimation errors, effectively avoiding filter divergence. Furthermore, in multi-path scenarios such as intersections, particle filtering can simultaneously maintain multiple pose assumptions until subsequent observations eliminate ambiguity. Compared to EKF, which can only choose a single assumption, particle filtering improves global optimality and reduces trajectory jump probability. Thus, this application accurately fuses multi-sensor data from vision, IMU, and GPS in a monocular panoramic imaging system, processing nonlinear motion models and non-Gaussian measurement noise, improving pose estimation accuracy and image stitching quality in complex scenes. Furthermore, this multi-sensor fusion method based on particle filtering can represent and track multimodal probability distributions, handle outliers and outlier measurements, and maintain stable positioning and stitching effects even in extreme cases such as visual failure and GPS signal loss.

[0058] Based on the aforementioned embodiments, since the key to particle filtering is calculating the observation likelihood of each particle, i.e., the probability that the observation conforms to the hypothesis of that particle, and then updating the particle weights, this application also discloses a specific calculation process for particle observation likelihood. The step of calculating the observation likelihood of each particle based on the current sensor data and the updated state vector includes: obtaining the feature matching result between the current bird's-eye view and the historical panoramic view, and calculating the visual observation likelihood of each particle using the Huber loss function based on the feature matching result and the updated state vector; calculating the positioning observation likelihood of each particle based on the horizontal observation data in the current second motion data and the horizontal position in the updated state vector; calculating the elevation observation likelihood of each particle based on the preset ground plane height and the elevation position in the updated state vector; determining the joint observation likelihood of each particle using the visual observation likelihood, the positioning observation likelihood, and the elevation observation likelihood; correspondingly, updating the weights of each particle using the observation likelihood includes: updating the weights of each particle using the joint observation likelihood and performing normalization processing to obtain the updated weights of each particle.

[0059] It is understandable that, such as Figure 3As shown, this application discloses a joint vision-GPS observation model, where the observation likelihood for each particle is a joint observation likelihood constructed based on visual observation likelihood, positioning observation likelihood, and elevation observation likelihood. When visual detection is effective (feature matching points > 30 and reprojection error < 5 pixels), the visual observation likelihood for each particle is calculated using the Huber loss function based on the feature matching results between the current bird's-eye view and the historical panoramic view, and the updated state vector. ; Given a particle state vector as Visual features observed at the current moment The likelihood probability. The positioning observation likelihood for each particle. It is then calculated based on the horizontal observation data in the current second motion data and the horizontal position in the updated state vector; Given a particle state vector as The horizontal observation data at the current time is observed below. The likelihood probability is calculated. Furthermore, to address the issues of monocular vision systems' inability to effectively observe absolute scales and the tendency for the vertical quadratic integral of the accelerometer to diverge, this application introduces Virtual Elevation Observation. When a vehicle travels on a road surface, it approximately satisfies the planar motion assumption, thus constructing a virtual elevation observation. (Assuming the vehicle is on a local horizontal plane). This embodiment requires calculating the elevation observation likelihood for each particle based on the preset ground plane height and the height position in the updated state vector. , Given a particle state vector as Virtual elevation data observed below The likelihood probability.

[0060] Furthermore, the visual observation likelihood, the localization observation likelihood, and the elevation observation likelihood are multiplied together to obtain the joint observation likelihood and update the particle weights: ; Update the weight of the i-th particle: ; in, Let be the weight of the i-th particle at time k+1. Let be the weight of the i-th particle at time k. Let be the joint observation likelihood of the i-th particle at time k+1.

[0061] Then the weights need to be normalized: .

[0062] In a specific implementation, the step of calculating the visual observation likelihood corresponding to each particle using the Huber loss function based on the feature matching result and the updated state vector includes: determining matching feature points corresponding to each target feature point in the current bird's-eye view based on the feature matching result in the historical panoramic image; reprojecting each matching feature point in the historical panoramic image to the bird's-eye view coordinate system according to the updated state vector corresponding to each particle to obtain the corresponding reprojected feature points; calculating the reprojection error between each target feature point in the current bird's-eye view and the corresponding reprojected feature point, and assigning feature point weights to each target feature point according to the ORB response intensity of the target feature point and the distribution position of the target feature point in the current bird's-eye view; weighting the reprojection error based on the feature point weights, and calculating the weighted reprojection error using the Huber loss function to obtain the error processing result; calculating the visual observation likelihood corresponding to each particle based on the error processing result and the visual noise variance; wherein, the visual noise variance is calculated based on the inlier rate corresponding to the feature matching result.

[0063] In this embodiment, M feature points are extracted from the current bird's-eye view, and M pairs of matching points are obtained by matching them with the historical panoramic view. For the i-th particle, assume its pose is... Matching feature points in the historical panoramic image By projecting the pose onto the bird's-eye view coordinate system, the corresponding reprojected feature points are obtained. The reprojection error is: ; Considering the varying reliability of different feature points, feature point weights are introduced. The reprojection error is weighted, and the feature point weights are determined by two parts: response intensity and distribution location, i.e.: ; The response intensity weight is: ; The ORB response value of the feature point.

[0064] The distribution location weights are: ; in, The distance from a feature point to its nearest neighbor is given by the given information. Pixels. This reduces the weight of feature points in clustered regions, encouraging evenly distributed feature matching.

[0065] The visual observation likelihood function is designed as a weighted Huber robust kernel function: ; in, Given a particle state vector as Visual features observed at the current moment The higher the likelihood probability, the better the pose hypothesis of the particle matches the visual observation. Let be the state vector of the i-th particle; This represents the visual observation data at the current moment; denoted as Variance of visual observation noise; M is the number of feature point pairs matched in the current frame; j is the index of the feature point. The weight of the j-th feature point; Huber loss function; The reprojection error of the i-th particle to the j-th matching feature point; This indicates that it is proportional to (the constant coefficient is ignored in the calculation).

[0066] It should be noted that in weakly textured scenes (such as underground parking garages), the visual likelihood model based on the Huber loss function remains robust even when outliers appear in visual measurements. The Huber function is defined as follows: ; Where e is the reprojection error. Using the Huber threshold, compared to quadratic error, the Huber function is less sensitive to outliers, thus improving robustness.

[0067] Visual noise variance Adaptive adjustment based on feature matching quality: ; in, The standard variance; For interior point ratio; It is a regulating factor.

[0068] When the interior point ratio is low, This application discloses a visual likelihood function based on feature quality, which weights matching features with confidence to suppress the influence of outliers. The weight of visual observations is increased, while the weight of mismatches is decreased to avoid interference from filters.

[0069] In a specific implementation, the step of calculating the positioning observation likelihood corresponding to each particle based on the horizontal observation data in the current second motion data and the horizontal position in the updated state vector includes: extracting the horizontal position from the updated state vector corresponding to each particle, and calculating the horizontal position error between the horizontal position and the horizontal observation data in the current second motion data; determining the position noise variance according to the horizontal accuracy factor of the preset positioning system, and calculating the positioning observation likelihood corresponding to each particle based on the horizontal position error and the position noise variance.

[0070] Understandably, GPS observation models provide global location observations. (Horizontal observation data; vertical data with poor accuracy is not used). For the i-th particle, the position error is: ; in, Let $\frac{i}{i}$ be the GPS horizontal position error of the $i$-th particle. The predicted horizontal position vector for the i-th particle ; This is the horizontal position vector observed by GPS at the current moment; It is the Euclidean norm (i.e., the vector length).

[0071] The GPS likelihood function adopts a two-dimensional isotropic Gaussian distribution: ; This assumes that the noise variance of GPS is the same in the horizontal x and y directions, which is consistent with the isotropic characteristics of a real GPS system.

[0072] Location noise variance Dynamically adjust based on GPS signal quality: ; in, is the baseline variance; HDOP is the level precision factor (value range 1-20). It is a regulating factor.

[0073] When HDOP > 10, the GPS signal quality is considered poor (e.g., obstructed by tall buildings, tunnel entrances). Increase or decrease the GPS weight.

[0074] In a specific implementation, the step of calculating the elevation observation likelihood corresponding to each particle based on the preset ground plane height and the height position in the updated state vector includes: extracting the height position from the updated state vector corresponding to each particle, and calculating the elevation error between the height position and the preset ground plane height; and calculating the elevation observation likelihood corresponding to each particle based on the elevation error and the preset virtual observation variance.

[0075] Understandably, this application addresses the limitations of monocular vision systems in effectively observing absolute scales and the tendency for accelerometer quadratic integrals in the vertical direction to diverge. This solution introduces virtual elevation observation. When a vehicle travels on a road surface, it approximately satisfies the planar motion assumption. The virtual elevation observation is then constructed. For the i-th particle, its elevation error is: ; in, The reference ground plane height (determined during initialization). The predicted height position of the i-th particle.

[0076] The likelihood function for elevation constraints is designed as follows: ; in, This is the virtual observation variance. This constraint is incorporated into the particle filter as a "soft constraint," allowing for minor undulations in the vehicle's position on bumpy roads, but strongly suppressing long-term height drift and ensuring geometric consistency of the panoramic projection transformation.

[0077] Furthermore, it should be noted that the adaptive weight adjustment for vision and GPS is achieved through noise variance. and This is automated. The collaborative workflow is as follows: First, sensor quality is assessed, specifically based on feature in-point ratio in the vision aspect. calculate GPS calculations are based on HDOP. ; During the likelihood function calculation phase, the larger the noise variance, the flatter the peak of the likelihood function, and the smaller its impact on particle weights. Specifically: Increasing the value of the visual likelihood function will flatten the visual likelihood function and reduce the influence of visual observation on the weights. Increasing the likelihood function will flatten the GPS likelihood function and weaken the influence of GPS observations on the weights. Through a dynamic weight balancing mechanism, when the quality of a sensor deteriorates, its noise variance increases, and its contribution to particle weights automatically decreases, causing the system to automatically rely on sensors with better quality. For example, in weakly textured scenes... Low leads to Increased HDOP leads to GPS-dominated weight updates; in cases of poor GPS signal, high HDOP results in... Increase, visually dominant weight update.

[0078] This noise variance-based adaptive mechanism eliminates the need for explicit calculation of weight coefficients, and achieves dynamic fusion of multiple sensors naturally through the shape of the likelihood function.

[0079] Building upon the foregoing embodiments, and considering that the number of particles directly affects estimation accuracy and computational efficiency—too few particles cannot accurately represent complex distributions, while too many particles impose a heavy computational burden—this application also discloses an adaptive particle management strategy to dynamically adjust the number of particles.

[0080] The step of adjusting the number of particles in the current particle set based on the effective particle count and the scene complexity determined using the statistical characteristics of the current sensor data to obtain an adjusted particle set includes: determining the scene complexity based on the statistical characteristics of the current sensor data, and determining a target particle count based on the effective particle count and the scene complexity; determining that the effective particle count is less than a preset quantity threshold; if the effective particle count is less than the preset quantity threshold, resampling the current particle set based on a preset hierarchical resampling algorithm to obtain a sampled particle set, and adjusting the number of particles in the sampled particle set based on the target particle count to obtain an adjusted particle set; if the effective particle count is not less than the preset quantity threshold, adjusting the number of particles in the current particle set based on the target particle count to obtain an adjusted particle set; wherein the number of particles in the sampled particle set is consistent with the number of particles in the current particle set.

[0081] In other words, this embodiment first determines the scene complexity C based on the statistical characteristics of the current sensor data, and then bases the determination on the effective number of particles. Determine the target particle number based on scene complexity: ; in, These are the upper and lower limits of the particle number. This is the particle degradation weight coefficient, which controls the influence of the degree of particle degradation on the target particle number. This is the scene complexity weight coefficient, which controls the weight of the influence of scene complexity on the number of target particles.

[0082] It is important to note that Used to reflect the degree of particle degradation The smaller the value (the more severe the degradation), the larger this term becomes, requiring more particles; The larger the number of particles, the more sensitive the system is to particle degradation, and the faster it will increase the number of particles to cope with degradation. Used to reflect scene complexity, a larger C indicates a more complex scene, requiring more particles to maintain estimation accuracy. The larger the particle size, the more sensitive the system is to scene complexity, and the more particle resources will be allocated in complex scenes.

[0083] For example, in a typical application scenario, a simple scenario (C=0.2, It requires approximately 180 particles. Complex scenarios (C=0.8, It requires approximately 372 particles. ).

[0084] Furthermore, particle degradation is an inherent problem in particle filtering. With iteration, weights gradually concentrate on a few particles, leading to… The degradation is a consequence of traditional resampling methods (such as systematic resampling), which, while addressing the degradation problem, lose particle diversity, causing the filter to converge to a local optimum prematurely. Therefore, this application also discloses a hierarchical resampling algorithm that preserves particle diversity during resampling.

[0085] In one specific implementation, when the effective number of particles A resampling mechanism is triggered at certain times to resample the current particle set to obtain a sampled particle set; where This is a preset threshold for the number of resamples. =0.5N. Furthermore, after resampling, adjustments need to be made based on the difference between the target particle count and the particle count N in the sampled particle set to obtain the adjusted particle set. It's important to note that the particle count in the sampled particle set is the same as the particle count in the current particle set, both being N.

[0086] In another specific implementation, if the number of effective particles is not less than a preset quantity threshold, there is no need to resample. Instead, the number of particles in the current particle set is adjusted directly based on the target number of particles to obtain the adjusted particle set.

[0087] As can be seen, this application dynamically adjusts the number of particles by combining scene complexity with the effective number of particles. Compared with the fixed number of particles scheme, it saves computing resources in simple scenes and ensures estimation accuracy in complex scenes.

[0088] In a specific implementation, the scene complexity is determined based on the statistical characteristics of the current sensor data, including: determining the motion complexity based on the standard deviation of acceleration and the standard deviation of angular velocity of the inertial measurement unit within a preset continuous frame; determining the visual complexity based on the total number of feature points extracted from the current bird's-eye view and a preset feature distribution uniformity; wherein the preset feature distribution uniformity is determined based on the number of feature points in each grid and the average number of feature points per grid in the current bird's-eye view; determining the positioning system complexity based on the horizontal accuracy factor of the preset positioning system; and performing a weighted calculation on the motion complexity, the visual complexity, and the positioning system complexity to obtain the scene complexity.

[0089] That is, such as Figure 4 As shown, the scenario complexity is defined in this application as follows: ; in, The motion complexity is determined by the standard deviation of IMU acceleration and the standard deviation of angular velocity. Visual complexity is characterized by the number of feature points and the uniformity of feature distribution; The complexity of the positioning system is determined by the horizontal accuracy factor of the positioning system; These are the weighting coefficients.

[0090] Specifically, motion complexity for: ; In the formula, These are the standard deviations of acceleration and angular velocity over the most recent 10 frames. This is the normalization constant.

[0091] Visual complexity for: ; Where M is the total number of feature points in the current frame (typically ranging from 50 to 300). The reference number of feature points is U, which represents the uniformity of feature distribution (range [0,1], close to 1 when uniform).

[0092] The feature distribution uniformity U is calculated using a grid partitioning method. The image is divided into... Count the number of feature points in each grid (e.g., 8×6=48 grids). Uniformity is defined as: ; in, This represents the average number of feature points per grid cell, i.e., the average number of feature points per grid cell. The standard deviation of the number of feature points; The coefficients are heterogeneous and reflect the degree of dispersion of the distribution.

[0093] When feature points are evenly distributed across all grid cells , When feature points are clustered in a few grid cells, As the value increases, U tends towards 0. A truncation step is added here to ensure U is non-negative. To avoid division by zero, when... Let U = 0.

[0094] Positioning system complexity for: ; Since the range of the tanh function is [0,1) and the sum of the weight coefficients is 1, the range of the overall complexity C is [0,1). C close to 0 indicates a simple scene (smooth motion, rich features, good GPS signal), and C close to 1 indicates a complex scene (intense motion, sparse features, poor GPS signal).

[0095] Furthermore, the step of resampling the current particle set based on a preset hierarchical resampling algorithm to obtain a sampled particle set includes: arranging the current particle set in descending order of weight to divide it into a preset high-weight layer, a preset medium-weight layer, and a preset low-weight layer; retaining all particles in the preset high-weight layer and generating new particles around the preset high-weight layer based on a preset noise strategy; resampling the particles in the preset medium-weight layer according to a preset weight ratio; and discarding all particles in the preset low-weight layer to obtain a sampled particle set based on all particles corresponding to the preset high-weight layer and all particles corresponding to the preset medium-weight layer.

[0096] That is, such as Figure 5 As shown, during resampling, the current particle set is divided into three layers according to weight based on a preset hierarchical resampling algorithm: a high-weight layer (the top 30% of particles by weight), a medium-weight layer (the middle 40% of particles by weight), and a low-weight layer (the bottom 30% of particles by weight). Each layer is resampled independently. High-weight layer: Retain all particles and generate new particles in a small area around them based on a preset noise strategy; Medium-weighted layer: Resampling is performed according to a preset weight ratio; Low-weight layers: Discard all of them and replace them with particles from high-weight and medium-weight layers.

[0097] Among them, for high-weight particles New particles are generated using a noise-adding strategy: ; noise The noise is zero-mean Gaussian, and the covariance matrix is: ; The scaling factor is denoted by Cov, which is the sample covariance of the particle set. This way, the newly generated particles can explore the surrounding state space while maintaining a similarity to the high-weighted particles.

[0098] In a specific implementation, the process of adjusting the number of particles in the current particle set or the sampled particle set based on the target particle number includes: if the target particle number is less than or greater than the particle number, then adjusting the particle number to the target particle number; if the target particle number is equal to the particle number, then keeping the particle number unchanged. That is, as... Figure 4 As shown, when When (particles need to be added), noise is added near high-weight particles to sample and generate new particles until... ;when When it's time to reduce particles, delete low-weight particles to reduce to a minimum. ,like If the error is less than 10, the current number of particles will be maintained.

[0099] As can be seen, this application maintains the quality and quantity of the particle set through the collaborative operation of adaptive particle number adjustment and hierarchical resampling. Specifically, the adaptive particle number adjustment strategy dynamically adjusts the particle number based on scene complexity, achieving a balance between computational efficiency and estimation accuracy; the hierarchical resampling algorithm maintains particle diversity while avoiding particle degradation. The collaborative mechanism between the two is as follows: During the evaluation phase, the effective particle count is calculated. The number of target particles is obtained by considering the scene complexity C. Perform a resampling judgment; if This triggers hierarchical resampling, resets particle weights while maintaining diversity; after quantity adjustment, resampling is performed according to... Adjustments are made based on the difference from the current particle number N: if Then, noise sampling is added near high-weight particles to increase the number of particles. ,like The removal of low-weight particles is reduced to ,like If the error is less than 10, the current number of particles is maintained. This cooperative mechanism ensures that the particle swarm can cope with degradation problems (through resampling) and adapt to changes in scene complexity (through dynamic adjustment of the number), achieving a dynamic balance between computational efficiency and estimation accuracy.

[0100] See Figure 6 As shown in the figure, this application discloses a panoramic imaging device, which includes: The particle filtering module 11 is used to acquire the current sensor data collected by the vehicle sensor, and based on the current sensor data, to update the state vector of each particle in the current particle set using a particle filtering algorithm to obtain the updated state vector; each particle carries a state vector and weights for representing the vehicle pose assumption. The weight update module 12 is used to calculate the observation likelihood of each particle based on the current sensor data and the updated state vector, and to update the weight of each particle using the observation likelihood. The particle number adjustment module 13 is used to determine the effective number of particles based on the updated weight corresponding to each particle in the current particle set, and adjust the number of particles in the current particle set based on the effective number of particles and the scene complexity determined by the statistical characteristics of the current sensor data, so as to obtain the adjusted particle set. The panoramic image generation module 14 is used to determine the optimal pose estimate based on the adjusted particle set, and to fuse the historical panoramic image and the bird's-eye view in the current sensor data using the optimal pose estimate to generate a new panoramic image.

[0101] As can be seen, this application uses a particle filter algorithm instead of the traditional EKF algorithm. It represents the posterior probability distribution of the vehicle pose using a set of particles carrying state vectors and weights, without relying on the assumption of a noisy Gaussian distribution or linearizing the nonlinear motion model. For outliers in visual measurements, the observation likelihood calculation effectively suppresses their interference with particle weights. Even in scenarios prone to non-Gaussian errors, such as weak textures or abrupt lighting changes, the accuracy of pose estimation is maintained, solving the stitching misalignment problem caused by discrepancies between the noise distribution assumption and reality in traditional methods. This application determines scene complexity based on the statistical characteristics of current sensor data and dynamically adjusts the particle set size based on the effective particle count. This adaptive particle count adjustment strategy resolves the contradiction between resource waste in simple scenes and insufficient accuracy in complex scenes in traditional fixed particle count schemes. Finally, the optimal pose estimate output by the particle filter is used to fuse the historical panoramic image and the current bird's-eye view, ensuring the geometric consistency of the projection transformation. By incorporating constraints from multi-sensor observation data, the continuity of pose estimation is maintained even in extreme scenarios such as visual failure. In this way, this application improves the accuracy of pose estimation and image stitching quality in complex scenes during panoramic imaging.

[0102] For a description of the features in the embodiments corresponding to the panoramic imaging device, please refer to the relevant descriptions in the embodiments corresponding to the aforementioned methods, which will not be repeated here.

[0103] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Specifically, it may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the panoramic imaging method performed by the electronic device disclosed in any of the foregoing embodiments.

[0104] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0105] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0106] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored on it include operating system 221, computer program 222 and data 223, etc., and the storage method can be temporary storage or permanent storage.

[0107] The operating system 221 manages and controls the various hardware devices and computer programs 222 on the electronic device 20 to enable the processor 21 to perform calculations and processing on the massive amounts of data 223 in the memory 22. The operating system 221 can be Windows, Unix, Linux, etc. The computer program 222, in addition to including a computer program capable of performing the panoramic imaging method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, may further include computer programs capable of performing other specific tasks. The data 223 may include data received by the electronic device from external devices, as well as data collected by its own input / output interface 25.

[0108] Furthermore, embodiments of this application also disclose a computer-readable storage medium storing a computer program, which, when loaded and executed by a processor, implements the panoramic imaging method steps disclosed in any of the foregoing embodiments.

[0109] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0110] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0111] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, compact disc read-only memory (CD-ROM), or any other form of storage medium known in the art.

[0112] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0113] The panoramic imaging method, apparatus, device, and storage medium provided by the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A panoramic imaging method, characterized in that, include: The system acquires current sensor data collected by vehicle sensors, and updates the state vector of each particle in the current particle set based on the current sensor data using a particle filtering algorithm to obtain the updated state vector; each particle carries a state vector and weights to represent the vehicle pose assumption. The observation likelihood for each particle is calculated based on the current sensor data and the updated state vector, and the weight of each particle is updated using the observation likelihood. The effective number of particles is determined based on the updated weight of each particle in the current particle set, and the number of particles in the current particle set is adjusted based on the effective number of particles and the scene complexity determined using the statistical characteristics of the current sensor data, so as to obtain the adjusted particle set. The optimal pose estimate is determined based on the adjusted particle set, and the optimal pose estimate is used to fuse the historical panoramic image and the bird's-eye view in the current sensor data to generate a new panoramic image.

2. The panoramic imaging method according to claim 1, characterized in that, The current sensor data includes the current bird's-eye view generated by projecting and transforming the image captured by the vehicle's monocular camera, the current first motion data captured by the inertial measurement unit, and the current second motion data captured by the preset positioning system; the state vector includes the vehicle's position and velocity in the ENU coordinate system, the attitude quaternion used to describe the transition from the vehicle coordinate system to the ENU coordinate system, and the zero bias value of the inertial measurement unit. Accordingly, the step of updating the state vector of each particle in the current particle set based on the current sensor data using a particle filtering algorithm to obtain the updated state vector includes: Based on the current first motion data collected by the inertial measurement unit, the state vector of each particle in the current particle set is updated using a particle filtering algorithm and a preset kinematic model to obtain the updated state vector.

3. The panoramic imaging method according to claim 2, characterized in that, The current first motion data includes the acceleration value measured by the accelerometer and the angular velocity value measured by the gyroscope; Accordingly, the process of updating the state vector of each particle in the current particle set based on the current first motion data collected by the inertial measurement unit and using a particle filtering algorithm and a preset kinematic model to obtain the updated state vector includes: The acceleration value is used to update the velocity and position in the state vector of each particle in the current particle set to obtain the updated velocity and updated position. The attitude quaternion in the state vector of each particle in the current particle set is updated based on the angular velocity value to obtain the updated attitude quaternion. The zero bias value in the state vector of each particle in the current particle set is updated based on the preset process noise to obtain the updated zero bias value. The updated state vector is obtained based on the updated velocity, the updated position, the updated attitude quaternion, and the updated zero bias.

4. The panoramic imaging method according to claim 2, characterized in that, The calculation of the observation likelihood for each particle based on the current sensor data and the updated state vector includes: Obtain the feature matching results between the current bird's-eye view and the historical panoramic view, and calculate the visual observation likelihood corresponding to each particle based on the feature matching results and the updated state vector using the Huber loss function; Calculate the positioning observation likelihood for each particle based on the horizontal observation data in the current second motion data and the horizontal position in the updated state vector. Based on the preset ground plane height and the height position in the updated state vector, calculate the elevation observation likelihood for each particle; The joint observation likelihood corresponding to each particle is determined using the visual observation likelihood, the positioning observation likelihood, and the elevation observation likelihood. Accordingly, updating the weights of each particle using the observed likelihood includes: The weights of each particle are updated using the joint observation likelihood and then normalized to obtain the updated weights of each particle.

5. The panoramic imaging method according to claim 4, characterized in that, The step of calculating the visual observation likelihood for each particle using the Huber loss function based on the feature matching result and the updated state vector includes: Based on the feature matching results, the matching feature points corresponding to each target feature point in the current bird's-eye view are determined in the historical panoramic view; Based on the updated state vector corresponding to each particle, the matching feature points in the historical panoramic image are reprojected onto the bird's-eye view coordinate system to obtain the corresponding reprojected feature points. Calculate the reprojection error between each target feature point and its corresponding reprojection feature point in the current bird's-eye view, and assign feature point weights to each target feature point based on the ORB response intensity of the target feature point and the distribution position of the target feature point in the current bird's-eye view; The reprojection error is weighted based on the feature point weights, and the Huber loss function is used to calculate the weighted reprojection error to obtain the error processing result. The visual observation likelihood for each particle is calculated based on the error processing result and the visual noise variance; wherein the visual noise variance is calculated based on the inlier rate corresponding to the feature matching result.

6. The panoramic imaging method according to claim 4, characterized in that, The calculation of the localization observation likelihood for each particle based on the horizontal observation data in the current second motion data and the horizontal position in the updated state vector includes: Extract the horizontal position from the updated state vector corresponding to each particle, and calculate the horizontal position error between the horizontal position and the horizontal observation data in the current second motion data; The position noise variance is determined based on the horizontal accuracy factor of the preset positioning system, and the positioning observation likelihood corresponding to each particle is calculated based on the horizontal position error and the position noise variance.

7. The panoramic imaging method according to claim 4, characterized in that, The step of calculating the elevation observation likelihood for each particle based on the preset ground plane height and the height position in the updated state vector includes: Extract the height position from the updated state vector corresponding to each particle, and calculate the elevation error between the height position and the preset ground plane height; Based on the elevation error and the preset virtual observation variance, the elevation observation likelihood corresponding to each particle is calculated.

8. The panoramic imaging method according to claim 2, characterized in that, The process of adjusting the number of particles in the current particle set based on the effective particle count and the scene complexity determined using the statistical characteristics of the current sensor data to obtain an adjusted particle set includes: The scene complexity is determined based on the statistical characteristics of the current sensor data, and the target particle number is determined based on the effective particle number and the scene complexity. It is determined that the number of effective particles is less than a preset quantity threshold; If the number of effective particles is less than the preset number threshold, the current particle set is resampled based on the preset hierarchical resampling algorithm to obtain the sampled particle set, and the number of particles in the sampled particle set is adjusted based on the target number of particles to obtain the adjusted particle set. If the number of effective particles is not less than the preset quantity threshold, then the number of particles in the current particle set is adjusted based on the target number of particles to obtain the adjusted particle set. The number of particles in the sampled particle set is the same as the number of particles in the current particle set.

9. The panoramic imaging method according to claim 8, characterized in that, The determination of scene complexity based on the statistical features of the current sensor data includes: The motion complexity is determined based on the standard deviation of acceleration and standard deviation of angular velocity of the inertial measurement unit within a preset continuous frame. The visual complexity is determined based on the total number of feature points extracted from the current bird's-eye view and the preset feature distribution uniformity; wherein, the preset feature distribution uniformity is determined based on the number of feature points in each grid in the current bird's-eye view and the average number of feature points in the grid. The complexity of the positioning system is determined based on the horizontal accuracy factor of the preset positioning system. The scene complexity is obtained by weighting the motion complexity, the visual complexity, and the positioning system complexity.

10. The panoramic imaging method according to claim 8, characterized in that, The step of resampling the current particle set based on a preset hierarchical resampling algorithm to obtain a sampled particle set includes: The current particle set is arranged in descending order of weight to divide it into a preset high-weight layer, a preset medium-weight layer, and a preset low-weight layer. All particles in the preset high-weight layer are retained, and new particles are generated around the preset high-weight layer based on a preset noise strategy. Particles in the preset weighted layer are resampled according to a preset weight ratio; Discard all particles in the preset low-weight layer to obtain a sampled particle set based on all particles corresponding to the preset high-weight layer and all particles corresponding to the preset medium-weight layer.

11. The panoramic imaging method according to claim 8, characterized in that, The process of adjusting the number of particles in the current particle set or the sampled particle set based on the target particle number includes: If the target number of particles is less than or greater than the number of particles, then the number of particles is adjusted to the target number of particles; If the number of target particles is equal to the number of particles, then the number of particles remains unchanged.

12. The panoramic imaging method according to any one of claims 1 to 11, characterized in that, The step of determining the optimal pose estimate based on the adjusted particle set, and fusing the optimal pose estimate with the historical panoramic image and the bird's-eye view from the current sensor data to generate a new panoramic image includes: Based on the state vectors and weights of each particle in the adjusted particle set, a weighted average is calculated as the optimal pose estimate of the vehicle at the current moment. Based on the optimal pose estimation transformation matrix, the historical panoramic image is projected onto the bird's-eye view coordinate system using the transformation matrix, and the projected historical panoramic image is fused with the current bird's-eye view based on a preset time weighting coefficient to generate a new panoramic image.

13. A panoramic imaging device, characterized in that, include: The particle filtering module is used to acquire the current sensor data collected by the vehicle sensors, and based on the current sensor data, to update the state vector of each particle in the current particle set using a particle filtering algorithm to obtain the updated state vector; each particle carries a state vector and weights representing the vehicle pose assumption. The weight update module is used to calculate the observation likelihood of each particle based on the current sensor data and the updated state vector, and to update the weight of each particle using the observation likelihood. The particle number adjustment module is used to determine the effective number of particles based on the updated weight corresponding to each particle in the current particle set, and adjust the number of particles in the current particle set based on the effective number of particles and the scene complexity determined by using the statistical characteristics of the current sensor data, so as to obtain the adjusted particle set. The panoramic image generation module is used to determine the optimal pose estimate based on the adjusted particle set, and to fuse the historical panoramic image and the bird's-eye view in the current sensor data using the optimal pose estimate to generate a new panoramic image.

14. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the panoramic imaging method as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, Used to store a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the panoramic imaging method as described in any one of claims 1 to 12.

Images