Panoramic imaging method, device, equipment and medium

Abstract

The invention discloses a panoramic imaging method and device, equipment and a medium, and relates to the technical field of automatic driving, and the method comprises the steps: carrying out the projection transformation of an original image collected by a vehicle camera, and generating an aerial view; determining the size of an adaptive target window according to the distance from each pixel to a central pixel in the aerial view, and performing optical flow estimation on each pixel in two adjacent frames of aerial views in the size of the target window based on vehicle motion information measured by an inertial measurement unit to obtain a dense optical flow field; extracting inner points from the dense optical flow field by using a preset improved RANSAC algorithm, and estimating global motion parameters based on an extracted inner point set; fusing the global motion parameters and motion parameters measured by the inertial measurement unit and the positioning system by using a Kalman filtering method to obtain fused motion parameters; and projecting the historical panoramic image to an aerial view coordinate system by using a homography matrix calculated based on fusion motion parameters, and splicing the historical panoramic image with the current aerial view to generate a new panoramic image.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a panoramic imaging method, apparatus, device, and medium. Background Technology

[0002] Monocular panoramic imaging technology has significant application value in the fields of vehicle-assisted driving and autonomous driving. Traditional monocular panoramic imaging methods typically rely on feature point matching techniques to achieve motion estimation and pose stitching between images. However, these methods have the following significant limitations: First, the feature point extraction and matching process is sparse, making it difficult to extract a sufficient number of stable feature points in regions with simple or repetitive textures, leading to matching failures. Second, the feature matching process is easily affected by dynamic objects, whose feature points may be mismatched, severely impacting the accuracy of motion estimation.

[0003] To overcome the limitations of feature point matching, optical flow technology has been introduced into the field of motion estimation. Optical flow can directly estimate the motion of each pixel in an image, providing dense motion field information without relying on sparse feature points. However, directly applying optical flow technology to monocular panoramic imaging still faces many challenges: optical flow estimates pixel-level motion information, while panoramic stitching requires image-level global motion parameters. A key issue is how to accurately and robustly extract global motion parameters representing the vehicle's own motion from a dense and potentially noisy optical flow field. Furthermore, pose estimation relying solely on optical flow information has poor robustness, and effectively fusing optical flow information with other sensor information is difficult to meet the real-time and accuracy requirements of vehicle panoramic imaging.

[0004] In summary, how to achieve stable, accurate, and efficient panoramic imaging based on optical flow technology is a problem that needs to be solved. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a panoramic imaging method, apparatus, device, and medium that can achieve stable, accurate, and efficient panoramic imaging based on optical flow technology. The specific solution is as follows: In a first aspect, this application discloses a panoramic imaging method, including: The raw images captured by the vehicle's camera are projected and transformed to generate a bird's-eye view; The size of the target window is determined based on the distance from each pixel in the bird's-eye view to the center pixel of the image. Within the target window size, optical flow estimation is performed on each pixel in two adjacent frames of bird's-eye view based on the vehicle motion information measured by the inertial measurement unit to obtain a dense optical flow field. The improved RANSAC algorithm is used to extract interior points from the dense optical flow field to obtain an interior point set, and global motion parameters are estimated based on the interior point set. The global motion parameters, the first motion parameters measured by the inertial measurement unit, and the second motion parameters measured by the positioning system are fused using the Kalman filtering method to obtain fused motion parameters. The homography matrix is ​​calculated based on the fused motion parameters, and the historical panoramic image is projected onto the bird's-eye view coordinate system using the homography matrix. Then, the projected historical panoramic image is stitched together with the current bird's-eye view to generate a new panoramic image.

[0006] Optionally, determining the target window size based on the distance from each pixel in the bird's-eye view to the center pixel of the image includes: Determine the center pixel of the bird's-eye view image, as well as the image width and height; Calculate the first Euclidean distance between each pixel and the center pixel of the image, and determine the maximum reference distance based on the image width and the image height; The window adjustment factor is determined based on the first Euclidean distance, the maximum reference distance, and the target window adaptive coefficient. The product of the window adjustment factor and the preset base window size is then determined as the target window size that adapts to each pixel.

[0007] Optionally, the step of estimating optical flow for each pixel in two adjacent bird's-eye view frames within the target window size based on vehicle motion information measured by the inertial measurement unit to obtain a dense optical flow field includes: Acquire the vehicle speed and angular velocity measured by the inertial measurement unit; Predict the prior optical flow of each pixel between two adjacent bird's-eye view frames based on the vehicle speed, the angular velocity, the coordinates of the center pixel of the image, and the bird's-eye view resolution. A first objective function containing optical flow constraint terms and prior constraint terms is constructed based on the objective prior weight coefficients; wherein, the optical flow constraint terms are constructed based on the optical flow constraint equation, and the prior constraint terms are constructed based on the prior optical flow; Within the target window size, the first objective function is solved using the weighted least squares method to obtain the optical flow vector of each pixel, and a dense optical flow field is obtained based on each optical flow vector.

[0008] Optionally, the panoramic imaging method further includes: The current scene type is determined based on the image texture features of the original image; Select target parameter combinations corresponding to the current scene type from a pre-calibrated offline parameter set; the target parameter combinations include the target window adaptive coefficient and the target prior weight coefficient, and the parameter set pre-stores parameter combinations corresponding to different scene types.

[0009] Optionally, the offline calibration process includes: During the offline phase, image sequences covering various scene types are acquired; The optical flow field of the image sequence is calculated using an improved PWC-Net network as a pseudo-true value; wherein, the improved PWC-Net network introduces a spatial attention mechanism in each layer of the feature pyramid, and the total loss function of the network is constructed based on L1 loss, gradient loss and spatial prior loss. With the goal of minimizing the error between the dense optical flow field corresponding to the image sequence and the pseudo-true value, and within a preset parameter search space, the optimal parameter combination is searched for different scene types based on a grid search method to obtain a parameter set.

[0010] Optionally, the step of extracting interior points from the dense optical flow field using a preset improved RANSAC algorithm includes: The optical flow confidence of each pixel in the dense optical flow field is obtained, and the spatial weight of each pixel is determined based on the distance between each pixel and the center pixel of the image; wherein, the optical flow confidence is determined based on the optical flow vector and structure tensor matrix corresponding to each pixel; The sampling probability of each pixel is determined based on the optical flow confidence and the spatial weight, and the target pixels for calculating the vehicle motion model are selected based on the sampling probability. For any target pixel, obtain the predicted optical flow value of the vehicle motion model for the target pixel, and calculate the optical flow error value between the optical flow vector corresponding to the target pixel and the predicted optical flow value. A target interior point determination threshold is determined that is compatible with any of the target pixels. If the optical flow error value is less than the target interior point determination threshold, then any of the target pixels is determined as an interior point of the vehicle motion model.

[0011] Optionally, determining the target in-point judgment threshold that matches any of the target pixels includes: Calculate the second Euclidean distance between any target pixel and the center pixel of the image, and determine the maximum reference distance based on the image width and image height of the bird's-eye view; A threshold adjustment factor is determined based on the second Euclidean distance, the maximum reference distance, and a preset distance coefficient. The product of the threshold adjustment factor and the preset basic interior point judgment threshold is determined as the target interior point judgment threshold that is adapted to any target pixel.

[0012] Optionally, estimating global motion parameters based on the set of inliers includes: Obtain the initial global motion parameters of the vehicle motion model. A second objective function containing the Huber loss function is constructed using the optical flow error values ​​of each point in the inlier set, and the initial global motion parameters are nonlinearly optimized using iterative reweighted least squares method to obtain the optimized global motion parameters.

[0013] Optionally, the step of fusing the global motion parameters, the first motion parameters measured by the inertial measurement unit, and the second motion parameters measured by the positioning system using the Kalman filtering method to obtain fused motion parameters includes: Construct a state vector that includes vehicle position, vehicle heading angle, vehicle speed, and vehicle angular velocity; The global motion parameters, the first motion parameters measured by the inertial measurement unit, and the second motion parameters measured by the positioning system are combined into a joint observation vector; wherein, the global motion parameters include rotation angles and translation vectors, the first motion parameters include vehicle acceleration and vehicle angular velocity, and the second motion parameters include vehicle position and vehicle velocity; The extended Kalman filter is executed in the prediction and update steps to estimate the state vector based on the joint observation vector and the target observation noise covariance matrix, so as to obtain the optimal estimate of the state vector, and the optimal estimate is used as the fused motion parameters.

[0014] Optionally, the target observation noise covariance matrix includes a first noise covariance matrix corresponding to the global motion parameters, a second noise covariance matrix corresponding to the inertial measurement unit, and a third noise covariance matrix corresponding to the positioning system; wherein, the first noise covariance matrix is ​​determined based on the average optical flow confidence of the dense optical flow field, the second noise covariance matrix is ​​determined based on the sensor accuracy of the inertial measurement unit, and the third noise covariance matrix is ​​determined based on the positioning accuracy of the positioning system.

[0015] Optionally, the process of determining the first noise covariance matrix based on the average optical flow confidence of the dense optical flow field includes: The dense optical flow field is optimized to obtain an optimized optical flow field, and the average optical flow confidence is calculated based on the optimized optical flow field. Calculate the initial noise covariance matrix based on the average optical flow confidence and the preset basic noise covariance matrix; The initial noise covariance matrix is ​​smoothed based on a preset smoothing coefficient and the smoothed noise covariance matrix corresponding to the historical frames to obtain the first noise covariance matrix after smoothing.

[0016] Optionally, optimizing the dense optical flow field to obtain the optimized optical flow field includes: The dense optical flow field is filtered based on a preset multi-level filtering mechanism to obtain a filtered optical flow field; wherein, the multi-level filtering mechanism includes, in sequence, a first filtering mechanism for maintaining the consistency of the optical flow of each pixel with the optical flow of neighboring pixels, a second filtering mechanism for verifying the difference between the global motion parameters and the motion parameters predicted by the inertial measurement unit, and a third filtering mechanism for smoothing the dense optical flow field. Based on the filtered optical flow field, a third objective function containing optical flow constraint terms and a smoothing term is constructed, and the third objective function is solved using an iterative method to obtain the optimized optical flow field; wherein, the optical flow constraint term is constructed based on the optical flow constraint equation, and the smoothing term is constructed based on the spatial gradient information of the optical flow.

[0017] Optionally, the step of stitching the projected historical panoramic image with the current bird's-eye view to generate a new panoramic image includes: Determine the overlapping area between the projected historical panoramic view and the current bird's-eye view; The fusion weight coefficient is calculated based on the difference in acquisition timestamps and the average value of the optical flow confidence of each pixel in the overlapping region; wherein, the difference in acquisition timestamps is the difference between the acquisition timestamp of the current bird's-eye view and the acquisition timestamp corresponding to the overlapping region in the historical panoramic image; The projected historical panoramic image and the current bird's-eye view are weighted and fused based on the fusion weight coefficient to obtain a fused image; The fused image is processed using multi-band fusion technology to eliminate seams, resulting in a new panoramic image.

[0018] Secondly, this application discloses a panoramic imaging device, comprising: The image conversion module is used to project and transform the raw images captured by the vehicle camera to generate a bird's-eye view; The optical flow estimation module is used to determine the size of the target window based on the distance from each pixel in the bird's-eye view to the center pixel of the image, and to perform optical flow estimation on each pixel in two adjacent frames of bird's-eye view based on the vehicle motion information measured by the inertial measurement unit within the target window size, so as to obtain a dense optical flow field. The parameter estimation module is used to extract interior points from the dense optical flow field using a preset improved RANSAC algorithm to obtain an interior point set, and to estimate global motion parameters based on the interior point set. The parameter fusion module is used to fuse the global motion parameters, the first motion parameters measured by the inertial measurement unit, and the second motion parameters measured by the positioning system using the Kalman filtering method to obtain fused motion parameters. The image stitching module is used to calculate the homography matrix based on the fused motion parameters, and to use the homography matrix to project the historical panoramic image onto the bird's-eye view coordinate system. Then, the projected historical panoramic image is stitched together with the current bird's-eye view to generate a new panoramic image.

[0019] Thirdly, this application discloses an electronic device, including: Memory, used to store computer programs; A processor is configured to execute the computer program to implement the steps of the aforementioned disclosed panoramic imaging method.

[0020] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the aforementioned disclosed panoramic imaging method.

[0021] As can be seen, this application generates a bird's-eye view by projecting and transforming the original image captured by the vehicle camera; determines the size of the target window based on the distance from each pixel in the bird's-eye view to the center pixel of the image, and estimates the optical flow of each pixel in two adjacent frames of the bird's-eye view within the target window size based on the vehicle motion information measured by the inertial measurement unit to obtain a dense optical flow field; extracts inliers from the dense optical flow field using a preset improved RANSAC algorithm to obtain an inlier set, and estimates global motion parameters based on the inlier set; fuses the global motion parameters, the first motion parameters measured by the inertial measurement unit, and the second motion parameters measured by the positioning system using the Kalman filtering method to obtain fused motion parameters; calculates a homography matrix based on the fused motion parameters, and projects the historical panoramic image onto the bird's-eye view coordinate system using the homography matrix, and then stitches the projected historical panoramic image with the current bird's-eye view to generate a new panoramic image.

[0022] Beneficial Effects: This application determines the appropriate target window size based on the distance from each pixel in the bird's-eye view to the image center. Simultaneously, it combines vehicle motion information from the inertial measurement unit (IMU) for optical flow estimation, introducing motion prior constraints. This combination of adaptive windowing and motion priors effectively solves the problem of unstable optical flow estimation in single-texture regions, resulting in a denser optical flow field that better reflects the actual vehicle motion. This application utilizes a pre-defined, improved RANSAC algorithm to extract interior points from the dense optical flow field, effectively filtering outomas caused by dynamic objects and lighting changes, selecting a set of interior points that conforms to the vehicle motion model. The global motion parameters estimated based on this set of interior points accurately reflect the vehicle's rotation and translation states, reducing motion parameter estimation errors and providing reliable foundational data for panoramic stitching. Furthermore, this application fuses the global motion parameters estimated by optical flow, the first motion parameters of the IMU, and the second motion parameters of the positioning system using Kalman filtering, comprehensively utilizing the short-term accuracy advantage of optical flow, the high-frequency response advantage of the IMU, and the long-term stability advantage of the positioning system. Multi-sensor data fusion effectively compensates for the limitations of a single sensor, maintaining high pose estimation accuracy even in complex scenes and avoiding stitching failures due to the malfunction of a single sensor. Finally, based on accurate fused motion parameters, the homography matrix is ​​calculated, enabling accurate projection of the historical panoramic image onto the current bird's-eye view coordinate system. A new panoramic image is then generated by stitching the projected historical panoramic image with the current bird's-eye view. This solution achieves stable, accurate, and efficient panoramic imaging based on optical flow technology. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0024] Figure 1 This is a flowchart of a panoramic imaging method disclosed in this application; Figure 2 This is a schematic diagram of the overall architecture of a panoramic imaging system disclosed in this application; Figure 3 This is a schematic diagram of a global motion parameter extraction process disclosed in this application; Figure 4 This is a schematic diagram of a multi-sensor fusion process disclosed in this application; Figure 5 This application discloses a flowchart of a multi-level outlier filtering process. Figure 6 This is a schematic diagram of the structure of a panoramic imaging device disclosed in this application; Figure 7 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0026] Currently, directly applying optical flow technology to monocular panoramic imaging still faces many challenges: optical flow estimates pixel-level motion information, while panoramic stitching requires image-level global motion parameters. A key issue is how to accurately and robustly extract global motion parameters representing the vehicle's own motion from a dense and potentially noisy optical flow field. Furthermore, pose estimation relying solely on optical flow information has poor robustness, and effectively fusing optical flow information with other sensor information is difficult to meet the real-time and accuracy requirements of vehicle panoramic imaging. Therefore, this application discloses a panoramic imaging method, apparatus, device, and medium that can achieve stable, accurate, and efficient panoramic imaging based on optical flow technology.

[0027] See Figure 1 and Figure 2 As shown in the figure, this application discloses a panoramic imaging method, which includes: Step S11: Perform projection transformation on the raw image captured by the vehicle camera to generate a bird's-eye view.

[0028] First, it should be noted that the vehicle camera in this embodiment mainly refers to a monocular camera, that is, a camera device consisting of only a single optical lens and image sensor. The monocular camera captures the latest images of the vehicle's surrounding environment in real time while the vehicle is in motion. Furthermore, after receiving the raw image (e.g., 1920×1080@30fps) captured by the monocular camera, this embodiment performs projection transformation processing on it using an image preprocessing model. Specifically, it uses camera intrinsic parameters, pitch angle, and distortion coefficients to convert the raw image into a bird's-eye view through distortion correction and perspective transformation; the bird's-eye view typically has a resolution of 800×800 pixels, covering an area of ​​approximately 20 meters × 20 meters around the vehicle. Further preprocessing can be performed on the bird's-eye view, including noise reduction and contrast enhancement, to improve the quality of subsequent optical flow estimation.

[0029] Step S12: Determine the appropriate target window size based on the distance from each pixel in the bird's-eye view to the center pixel of the image, and perform optical flow estimation on each pixel in two adjacent frames of the bird's-eye view within the target window size based on the vehicle motion information measured by the inertial measurement unit to obtain a dense optical flow field.

[0030] This embodiment also discloses an optical flow estimation module. After obtaining the bird's-eye view, this module determines the size of the appropriate target window based on the distance from each pixel in the bird's-eye view to the image center. Simultaneously, within this target window size, it combines vehicle motion information measured by the inertial measurement unit (IMU) to perform optical flow estimation on each pixel in two adjacent bird's-eye view frames, introducing motion prior constraints. This combination of adaptive window strategy and motion prior effectively solves the problem of unstable optical flow estimation in single-texture regions, and the generated dense optical flow field better matches the actual vehicle motion state. Specifically, the two adjacent bird's-eye view frames refer to the current bird's-eye view and the previous bird's-eye view. The current bird's-eye view is obtained by projecting and transforming the current frame image captured by the vehicle camera, and the previous bird's-eye view is obtained by projecting and transforming the previous frame image captured by the vehicle camera.

[0031] It should be noted that this application essentially improves the Lucas-Kanade algorithm because the traditional Lucas-Kanade algorithm has shortcomings in bird's-eye view scenarios: vehicle motion in bird's-eye view images mainly occurs in the central region of the image, with less motion in the edge regions; traditional algorithms use the same processing strategy for all regions; ground textures in bird's-eye view images are highly repetitive, easily leading to ambiguous matching; and traditional algorithms do not consider prior knowledge of vehicle motion. Therefore, this application improves the Lucas-Kanade algorithm for bird's-eye view scenarios, thereby utilizing the improved Lucas-Kanade algorithm for real-time dense optical flow field estimation. Specifically, it addresses the problems of optical flow estimation in bird's-eye view scenarios by employing spatially adaptive window size and prior constraints on vehicle motion.

[0032] Step S13: Extract interior points from the dense optical flow field using the preset improved RANSAC algorithm to obtain an interior point set, and estimate global motion parameters based on the interior point set.

[0033] It should be noted that this embodiment also discloses a global motion parameter extraction module, used to extract global motion parameters based on a dense optical flow field. The global motion parameters include rotation angles and translation vectors. It is assumed that the motion between bird's-eye view images can be described using a rigid body transformation model. ; in, These are the pixel coordinates in the first frame; R represents the corresponding pixel coordinates in the second frame; R is a 2×2 rotation matrix. ; It is a translation vector. The angle is the rotation angle.

[0034] This rigid body transformation model describes the motion relationships between bird's-eye views.

[0035] For small-angle rotation ( (Radians, approximately 5.7 degrees) can be utilized , For small-angle rotations, linearization can be performed: .

[0036] In this embodiment, the traditional RANSAC (Random Sample Consensus) algorithm has shortcomings in bird's-eye view scenarios: low random sampling efficiency, easy sampling of outliers; lack of utilization of prior knowledge of vehicle motion; and a fixed threshold for interior point judgment, making it unreasonable to use the same standard for different areas. Therefore, this embodiment improves the RANSAC algorithm for bird's-eye view scenarios. The improved RANSAC algorithm extracts interior points from a dense optical flow field, effectively filtering outliers caused by dynamic objects and lighting changes, selecting a set of interior points that conforms to the vehicle motion model. The global motion parameters estimated based on this set of interior points accurately reflect the vehicle's rotation and translation states, reducing the error in motion parameter estimation and providing reliable basic data for panoramic stitching.

[0037] Furthermore, this application directly estimates dense optical flow fields, with each pixel having motion information, avoiding the limitations of feature point extraction and matching, and can still work stably in scenarios with rich textures but sparse or difficult feature points.

[0038] Step S14: Use the Kalman filter method to fuse the global motion parameters, the first motion parameters measured by the inertial measurement unit, and the second motion parameters measured by the positioning system to obtain fused motion parameters.

[0039] In this embodiment, Kalman filtering is used to fuse the global motion parameters estimated by optical flow, the first motion parameters of the inertial measurement unit (IMU), and the second motion parameters of the positioning system (such as GPS) to obtain fused motion parameters. This approach comprehensively utilizes the short-term accuracy advantage of optical flow, the high-frequency response advantage of the inertial measurement unit, and the long-term stability advantage of the positioning system. In other words, this application, through its multi-sensor fusion module, can effectively compensate for the limitations of a single sensor, maintaining high pose estimation accuracy even in complex scenarios and avoiding stitching failures due to the failure of a single sensor.

[0040] Step S15: Calculate the homography matrix based on the fused motion parameters, and use the homography matrix to project the historical panoramic image onto the bird's-eye view coordinate system. Then, stitch the projected historical panoramic image with the current bird's-eye view to generate a new panoramic image.

[0041] In this embodiment, the homography matrix is ​​calculated based on accurate fused motion parameters, thereby enabling accurate projection of the historical panoramic image onto the current bird's-eye view coordinate system. It can be understood that the fused motion parameters are the optimal estimates of the state vector output after Kalman filtering, thus yielding the optimized position. ; in, Let be the optimal estimate of the state vector at time k. For vehicle location, For the vehicle's heading angle, For vehicle speed, This represents the vehicle's angular velocity.

[0042] The specific parameter used is the difference between the state vectors of adjacent frames, i.e., the change in heading angle. and the changes in position in the x and y directions ;in, , , , k and k -1 respectively corresponds to the first k Time and the k -1 hour.

[0043] These three parameters are used to construct the 2D rigid body transformation matrix. Then, the homography matrix H is calculated to achieve image stitching. Understandably, this application requires calculating the transformed homography matrix H based on the fused motion parameters to transform the historical panoramic image coordinate system to the current bird's-eye view coordinate system. Assume the coordinates of a point in the historical panoramic image are... The coordinates of the corresponding point in the current bird's-eye view are: Then its transformation relationship is: ; Where H is a 3×3 homography matrix, ; This is the scale matrix of the bird's-eye view, used to convert physical space coordinates into image pixel coordinates: ; Where s is the resolution of the bird's-eye view; The inverse of the scaling matrix is ​​used to restore the image pixel coordinates to physical space coordinates; The 2D rigid body transformation matrix (3×3 homogeneous form) in physical space describes the motion of the vehicle in the physical world: ; in, This represents the change in the vehicle's position in the physical coordinate system. This represents the change in heading angle.

[0044] The formula clarifies the transformation process: first, the pixel coordinates are restored to physical coordinates. Applying rigid body motion in physical space ( Finally, convert it back to pixel coordinates. This completely avoids the concept of perspective camera intrinsic parameter K, accurately reflecting the map mapping characteristics of bird's-eye view.

[0045] In this way, the embodiments of this application can use the calculated homography matrix H to project the historical panoramic image onto the current bird's-eye view coordinate system. Furthermore, this application uses a panoramic image stitching module to stitch the projected historical panoramic image with the current bird's-eye view to generate a new panoramic image.

[0046] In a specific implementation, the above-mentioned method of stitching the projected historical panoramic image with the current bird's-eye view to generate a new panoramic image includes: determining the overlapping area between the projected historical panoramic image and the current bird's-eye view; calculating a fusion weight coefficient based on the acquisition timestamp difference and the average value of the optical flow confidence of each pixel in the overlapping area; wherein the acquisition timestamp difference is the difference between the acquisition timestamp of the current bird's-eye view and the acquisition timestamp corresponding to the overlapping area in the historical panoramic image; performing weighted fusion of the projected historical panoramic image and the current bird's-eye view based on the fusion weight coefficient to obtain a fused image; and using multi-band fusion technology to perform seam elimination processing on the fused image to obtain a new panoramic image.

[0047] In this embodiment, the overlapping area between the projected historical panoramic image and the current bird's-eye view is first calculated. Then, a weighted average is used to fuse the overlapping area. The weights are based on the acquisition timestamp and optical flow confidence. The fusion weight coefficients corresponding to the historical panoramic image are as follows: The calculation formula is: ; The fusion weight coefficient corresponding to the current bird's-eye view for: ; in, This is the fusion weight coefficient for the historical panoramic image, with a value range of (0,1). It is a natural exponential function. This is the time decay factor, with a value of 0.1, which controls the rate at which the weight of the historical panoramic image decays over time. This is the timestamp of the current bird's-eye view capture. This refers to the data collection timestamp corresponding to the overlapping area in the historical panoramic image. This represents the average optical flow confidence level for each pixel in the overlapping region.

[0048] So, the merged pixel values The calculation formula is: ; in, Let be the pixel value at position (x, y) of the projected historical panoramic image. This represents the pixel value at position (x, y) in the current bird's-eye view.

[0049] It should be noted that in this embodiment, the weighting principle is that older historical images have lower weights (time decay), and lower optical flow confidence indicates poor registration reliability, making the current image more reliable. Furthermore, for non-overlapping regions, the image covering that region can be used directly.

[0050] Furthermore, this embodiment also utilizes multi-band fusion technology to eliminate seams in the fused image to obtain a new panoramic image. It is understood that even after weighted fusion, visible seams may still appear at the stitching boundary due to differences in illumination, exposure variations, or minor registration errors. Multi-band fusion eliminates these seams by employing different mixing strategies for different frequency components of the image.

[0051] First, construct the Laplacian pyramid and then merge the images. And the current bird's-eye view Decomposed into L layers respectively: ; Where L represents the number of pyramid layers, typically 4-6 layers; l For layer index, ; For the first l Layered Laplacian images contain detailed information (high-frequency components) at that scale. The topmost (Lth layer) Gaussian image contains overall brightness information (low-frequency components).

[0052] Simultaneously construct the mask pyramid The mask M defines the blending weights for the overlapping regions, smoothly transitioning from the boundary inwards. Then, each layer is fused separately. ; in, For the first l Laplacian image after layer fusion The first of the historical panorama l Layered Laplacian image, The current bird's-eye view l Layered Laplacian image, For the first l Layer blending mask.

[0053] Low-frequency layer ( l The mask transition area is wider (larger), achieving a smooth transition of illumination; the high-frequency layer ( l The smaller mask transition area is narrower, preserving sharp detail edges. Finally, the image is reconstructed layer by layer from the top of the pyramid to obtain the multi-band fused image. This eliminates visible seams at the splicing boundaries.

[0054] Finally, the multi-band fused image is used as a new panoramic image. This data is used for stitching in the next frame. Through continuous inter-frame fusion, a complete panoramic image is finally output.

[0055] As can be seen, this application determines the appropriate target window size based on the distance from each pixel in the bird's-eye view to the image center. Simultaneously, it combines vehicle motion information from the inertial measurement unit (IMU) for optical flow estimation, introducing motion prior constraints. This combination of adaptive windowing and motion priors effectively solves the problem of unstable optical flow estimation in single-texture regions, resulting in a denser optical flow field that better reflects the actual vehicle motion. This application utilizes a pre-defined, improved RANSAC algorithm to extract interior points from the dense optical flow field, effectively filtering outomas caused by dynamic objects and lighting changes, selecting a set of interior points that conforms to the vehicle motion model. The global motion parameters estimated based on this set of interior points accurately reflect the vehicle's rotation and translation states, reducing motion parameter estimation errors and providing reliable foundational data for panoramic stitching. Furthermore, this application fuses the global motion parameters estimated by optical flow, the first motion parameters of the IMU, and the second motion parameters of the positioning system using Kalman filtering, comprehensively utilizing the short-term accuracy advantage of optical flow, the high-frequency response advantage of the IMU, and the long-term stability advantage of the positioning system. Multi-sensor data fusion effectively compensates for the limitations of a single sensor, maintaining high pose estimation accuracy even in complex scenes and avoiding stitching failures due to the malfunction of a single sensor. Finally, based on accurate fused motion parameters, the homography matrix is ​​calculated, enabling accurate projection of the historical panoramic image onto the current bird's-eye view coordinate system. A new panoramic image is then generated by stitching the projected historical panoramic image with the current bird's-eye view. This solution achieves stable, accurate, and efficient panoramic imaging based on optical flow technology.

[0056] Based on the foregoing embodiments, this application discloses an adaptive determination process for the target window size. Specifically, determining the appropriate target window size based on the distance from each pixel in the bird's-eye view to the center pixel of the image may include: determining the center pixel of the bird's-eye view, and determining the image width and image height of the bird's-eye view; calculating a first Euclidean distance between each pixel and the center pixel, and determining a maximum reference distance based on the image width and image height; determining a window adjustment factor based on the first Euclidean distance, the maximum reference distance, and the target window adaptive coefficient, and determining the product of the window adjustment factor and a preset base window size as the target window size adapted to each pixel.

[0057] In other words, this embodiment primarily adjusts the window size adaptively based on the distance from the pixel to the center pixel of the image. The main purpose is to use a larger window to cover the greater displacement in the central region of the image where the movement is large, and a smaller window to cover the smaller movement in the edge region. Therefore, the formula for calculating the target window size is: ; In the formula, The base window size is set to 15 pixels. The window adjustment factor is primarily based on the first Euclidean distance. Maximum reference distance and target window adaptive coefficients To be determined jointly.

[0058] in, For each pixel With the center pixel of the image The Euclidean distance between them The maximum reference distance is half the length of the image diagonal, where W is the image width and H is the image height. The default initial value is 0.5, and it is dynamically selected based on the offline calibration results and the current scene type during actual operation.

[0059] This results in a larger window in the central region and a smaller window in the edge regions, thus adapting to the different motion characteristics of different regions. Subsequently, for each pixel... With it as the center and its size as Optical flow estimation is performed within a window, and all pixels within the window are denoted as the set. .

[0060] Furthermore, the step of estimating the optical flow of each pixel in two adjacent bird's-eye view frames based on the vehicle motion information measured by the inertial measurement unit within the target window size to obtain a dense optical flow field includes: acquiring the vehicle speed and angular velocity measured by the inertial measurement unit; predicting the prior optical flow of each pixel between two adjacent bird's-eye view frames based on the vehicle speed, the angular velocity, the coordinates of the image center pixel, and the bird's-eye view resolution; constructing a first objective function containing optical flow constraint terms and prior constraint terms based on the target prior weight coefficients; wherein the optical flow constraint terms are constructed based on the optical flow constraint equation, and the prior constraint terms are constructed based on the prior optical flow; solving the first objective function using the weighted least squares method within the target window size to obtain the optical flow vector of each pixel, and obtaining a dense optical flow field based on each optical flow vector.

[0061] In other words, this embodiment uses vehicle motion information measured by the IMU to constrain optical flow estimation; the IMU provides vehicle speed. and vehicle angular velocity This allows for the prediction of optical flow. Specifically, for pixels in a bird's-eye view... Its prior optical flow between two adjacent bird's-eye view frames is: ; ; Where Δt is the inter-frame time interval; s For a bird's-eye view resolution of 800×800 pixels covering a 20m×20m bird's-eye view, s=0.025m / pixel; The coordinates of the center pixel of the image; and Corresponding to x direction and y Prior optical flow in the direction.

[0062] Furthermore, this application also needs to construct the objective function corresponding to the optical flow estimation, which is specifically constructed based on the objective prior weight coefficients and includes optical flow constraint terms and prior constraint terms; wherein, the optical flow constraint terms are constructed based on the optical flow constraint equation, and the prior constraint terms are constructed based on the prior optical flow.

[0063] First, let's briefly introduce the basic principles of optical flow. Optical flow is the instantaneous velocity of a pixel in an image. Suppose a point in the image... The brightness at time t is At that point in time Move to The brightness remains unchanged, that is: ; in, Let be the coordinates of the pixel in the image, and t be the time. For pixels The gray value at time t, For pixels in time intervals Displacement within, The time interval (frame) is usually 1 frame.

[0064] Performing a Taylor expansion on the right-hand side, ignoring higher-order terms, we get: ; The optical flow constraint equation is obtained by refining the equation, which describes the pixel motion relationship under the assumption of constant image brightness: ; in, Let be the gradient of the image in the x-direction. Let be the gradient of the image in the y-direction. The gradient of the image in the time direction. Let x be the optical flow velocity in the x-direction. Let y be the optical flow velocity in the y direction.

[0065] The specific expression for the first objective function is: ; in, For pixels The adaptive window size; u is the value to be solved. x Directional optical flow; v For the solution to be found y Directional optical flow; i For window Pixel index within; Let be the image grayscale gradient of the i-th pixel within the window in the x-direction; Let be the image grayscale gradient of the i-th pixel in the y-direction within the window; Let be the image grayscale gradient of the i-th pixel within the window in the time direction; The target prior weight coefficient has a default initial value of 0.1. In actual operation, it is dynamically selected based on the offline calibration results and the current scene type (the calibration search range is [0.01, 0.5]). and Corresponding to x direction and y Prior optical flow in the direction.

[0066] Understandably, the objective function consists of two parts. For optical flow constraints, based on the assumption of constant brightness, As a prior constraint, it constrains the optical flow estimation to approach the IMU prediction. This weight balances the optical flow constraint and the prior constraint: when the scene texture is rich, the optical flow constraint dominates; when the texture is simple, the prior constraint provides stability.

[0067] Finally, within the target window size, the first objective function is solved using weighted least squares to obtain the optical flow vectors of each pixel, and a dense optical flow field is obtained based on these vectors. Specifically, the optimization solution uses weighted least squares, applying the first objective function to... u and v Taking the partial derivatives and setting them to zero, we obtain a system of linear equations: ; The system of linear equations is of the form Ax=b, where the coefficient matrix A (a 2×2 matrix) contains: This represents the sum of squared gradients in the x-direction of all pixels within the window, plus the target prior weight coefficient. , This represents the sum of the products of the x-direction gradient and the y-direction gradient of all pixels within the window. This represents the sum of squared gradients in the y-direction of all pixels within the window, plus the target prior weight coefficient. , target window size The abbreviation of .

[0068] Unknown vector x middle, u For the solution to be found x directional optical flow, v For the solution to be found y Directional optical flow; In the constant vector b, Indicates the window x The sum of the products of the directional gradient and the temporal gradient, minus the prior weights. x The product of directional prior optical flow. Indicates the window y The sum of the products of the directional gradient and the temporal gradient, minus the prior weights. y The product of directional prior optical flow.

[0069] Solving this system of linear equations yields the optical flow. , Adding diagonal elements to the coefficient matrix improves numerical stability and also enables prior prediction of the solution direction. Offset.

[0070] It should be noted that the method in this embodiment further includes: determining the current scene type based on the image texture features of the original image; selecting a target parameter combination corresponding to the current scene type from a pre-calibrated offline parameter set; the target parameter combination includes the target window adaptive coefficient and the target prior weight coefficient, and the parameter set pre-stores parameter combinations corresponding to different scene types. That is, in the offline stage or calibration mode, this application will adjust key parameters in the improved Lucas-Kanade algorithm, such as the target window adaptive coefficient. and target prior weight coefficient Optimization and dynamic calibration are performed to obtain parameter combinations corresponding to different scene types. Therefore, during online operation, the system determines the current scene type based on the image texture features of the original image, and then automatically selects the corresponding target parameter combination from the parameter set to obtain the target window adaptive coefficient and the target prior weight coefficient, thereby achieving scene classification and parameter mapping.

[0071] The offline calibration process specifically includes: in the offline stage, acquiring image sequences covering multiple scene types; calculating the optical flow field of the image sequences using an improved PWC-Net network as a pseudo-true value; wherein the improved PWC-Net network introduces a spatial attention mechanism in each layer of the feature pyramid, and the total network loss function is constructed based on L1 loss, gradient loss, and spatial prior loss. With the goal of minimizing the error between the dense optical flow field corresponding to the image sequence and the pseudo-true value, and within a preset parameter search space, the optimal parameter combination is searched for different scene types based on a grid search method to obtain a parameter set.

[0072] Understandably, the system uses an offline calibration mechanism to optimize key parameters of the improved Lucas-Kanade algorithm using pseudo-true values ​​generated by the PWC-Net network. The calibration process is as follows: 1. In the offline stage or system calibration mode, acquire image sequences covering a variety of typical road conditions, including scenes with rich textures (such as urban roads), scenes with simple textures (such as highways), and different vehicle speed ranges.

[0073] 2. For the acquired image sequences, a high-precision optical flow field is calculated using an improved PWC-Net network as a pseudo-true value. .

[0074] 3. Output using the improved Lucas-Kanade algorithm With false truth value The endpoint error (EPE) between nodes is used as the optimization objective. ; Where EPE is the endpoint error, representing the average pixel deviation of the optical flow estimation; N is the number of effective pixels involved in the calculation. i For pixel index, ; For the improved Lucas-Kanade algorithm, the first... i estimated per pixel x Directional optical flow; For the improved Lucas-Kanade algorithm, the first... i estimated per pixel y Directional optical flow; The pseudo-real values ​​generated for PWC-Net are the first i 1 pixel x Directional optical flow; The pseudo-real values ​​generated for PWC-Net are the first i Optical flow in the y-direction of pixels.

[0075] 4. Employ a grid search method to search for the optimal parameter combination within a pre-defined parameter search space, for the window adaptive coefficient. The search scope is The step size is 0.1; for prior weights The search scope is The step size is 0.02. Optimal parameters are searched for for each road condition scenario: ; in, The optimal window adaptation coefficient is used to control the degree to which the window size changes with the distance from the pixel to the image center; The optimal prior weights (dimensionless) are used to balance the relative importance of optical flow constraints and IMU prior constraints. Indicates that under given parameters and The endpoint error is calculated below.

[0076] 5. Divide the scene into multiple scene types based on image texture features, and store the corresponding optimal parameter combination for each scene type.

[0077] 6. The system runs a calibration process periodically (e.g., weekly or after accumulating a certain mileage) to update the optimal parameters for each scenario category based on recent driving data, thereby achieving dynamic adjustment of parameters and ensuring that the system always maintains optimal performance under different road conditions.

[0078] It should also be noted that the traditional PWC-Net network has shortcomings in bird's-eye view scenarios: the network structure is generic and not optimized for the geometric characteristics of bird's-eye views; the loss function does not consider the differences in importance between different regions in the bird's-eye view; and it does not utilize prior knowledge of vehicle motion. Therefore, this application improves PWC-Net for bird's-eye view scenarios: Network structure optimization: Spatial attention module; Building upon PWC-Net, a spatial attention mechanism is added to each layer of the feature pyramid, causing the network to focus on the central region of the image (the area where vehicle motion mainly occurs). The spatial attention module is inserted after the feature extraction network and before the cost volume construction.

[0079] For feature maps Introduce distance graph Each position The value is: ; Where F is the feature map of the current pyramid layer; H and W are the height and width of the feature map, respectively; and C is the number of channels in the feature map. For position The distance value ranges from [0,1], with the center being 1 and the edges approaching 0; The coordinates of the image center; It is half the length of the image's diagonal.

[0080] Attention weights are calculated by concatenating the distance map and the feature map using a 1×1 convolution: ; Where A is the attention weight map ( The value at each position is between (0, 1); The sigmoid activation function maps the output to the (0,1) interval; This is a 1×1 convolution operation used for feature fusion along the channel dimension. To stitch the feature map F and the distance map D along the channel dimension, we obtain (C+1) channels.

[0081] The final output is: ; in, For the enhanced feature map, This is element-wise multiplication (Hadamard product). The attention-weighted features are combined with the original features F to form residual connections, ensuring the flow of information.

[0082] The attention module guides the network to focus more on the central region of the image through a distance map. The enhanced feature map is then used for subsequent cost volume construction and optical flow decoding. Specifically, the enhanced feature map output by the spatial attention module... The predicted optical flow is obtained through the following steps: using... The correlation between features of two frames of images is calculated to construct the cost volume. The cost volume is then processed by a convolutional decoder to output the predicted optical flow. Each pyramid layer outputs the predicted optical flow at the corresponding resolution to achieve multi-scale output.

[0083] Furthermore, this application improves the loss function, which is mainly used during the network training phase. By comparing the difference between the predicted optical flow and the ground truth optical flow, the loss value is calculated, and the parameters of the entire network (including the convolutional parameters of the spatial attention module) are optimized through backpropagation. After training, the network parameters are fixed, and only forward propagation is performed during the inference phase to output the predicted optical flow. In short, the spatial attention module improves the quality of feature extraction, thereby enhancing the accuracy of predicted optical flow, and the loss function optimizes the parameters of these modules during training.

[0084] In other words, this application discloses a multi-scale weighted loss function by improving the loss function, taking into account the spatial characteristics of bird's-eye view: ; in, This is the total loss value (dimensionless), used for backpropagation during network training; This represents the summation over all levels of the pyramid (from level 0 to level L); For pyramid level indexing, L is the index of the highest level of the pyramid, with a value of 6, meaning there are 7 pyramid levels in total. For the first l Layer weights This makes the coarser layers (lower resolution) have a greater weight; For the first l Weighted L1 loss of the layer; For the first l Weighted gradient loss of the layer; For the first l Spatial prior loss of layers; This is the gradient loss weight, with a value of 0.5; This is the spatial prior loss weight, with a value of 0.2.

[0085] The weighted L1 loss formula is as follows: ; in, For the first l Total number of effective pixels in the layer i For pixel index, For the firsti Spatial weights of each pixel For the network in the first l Layer to the first i The x-direction optical flow predicted for each pixel. For the network in the first Layer to the first Predicted per pixel y directional optical flow, For the first l Layer i 1 pixel x Directional truth optical flow For the first l Layer i 1 pixel y Directional truth optical flow The formula for weighted gradient loss is: ; in, For the first l Weighted gradient loss of layers, Predicting optical flow for networks In the i Spatial gradient at each pixel ; Predicting optical flow for networks In the i Spatial gradient at each pixel True value optical flow In the i Spatial gradient at each pixel True value optical flow In the i Spatial gradient at each pixel.

[0086] The formula for spatial prior loss is: ; in, For the first l Spatial prior loss of layers, For the prediction based on IMU data i 1 pixel x Directional a priori optical flow For the prediction based on IMU data i Prior optical flow in the y-direction of each pixel Spatial weight formula: ; in, represents the spatial weight of the i-th pixel, with a value range of (0,1]. Let be the Euclidean distance from the i-th pixel to the image center. The standard deviation of the Gaussian distribution is taken as 1 / 3 of the image width, which controls the weight decay rate.

[0087] Spatial prior optical flow is predicted by combining IMU data with a motion model: ; ; in, Let x be the vehicle's velocity in the x-direction. Let be the vehicle's speed in the y-direction, and s be the resolution of the bird's-eye view. The time interval between adjacent frames. Let ω be the vehicle's angular velocity. Let be the coordinates of the i-th pixel. These are the coordinates of the image center.

[0088] The multi-scale loss function in this application makes the network focus more on the central region of the image through spatial weighting, the gradient loss ensures the smoothness of the optical flow field, and the spatial prior loss uses vehicle motion priors to improve estimation accuracy.

[0089] As can be seen, this application discloses a dual-engine optical flow estimation scheme with "real-time-calibration" collaboration. The real-time engine employs an improved Lucas-Kanade algorithm for real-time operation on an automotive embedded platform; the calibration engine uses an improved PWC-Net deep learning network for offline parameter adaptation and periodic system calibration. Specifically, considering the computational resource limitations of the automotive embedded platform, the improved Lucas-Kanade algorithm (average processing time approximately 20ms / frame) is used for real-time dense optical flow field estimation during actual online operation. Spatial adaptation and motion prior constraint optimization are performed for bird's-eye view scenarios. Specifically, the spatial adaptation window uses a larger window in the image center region to handle large displacement motions; vehicle motion prior constraints utilize IMU data to improve the stability of optical flow estimation. Furthermore, to overcome the limitations of fixed parameters in traditional algorithms, an improved PWC-Net network is introduced as an offline calibrator (processing time approximately 80ms / frame). Its high-precision characteristics are used to generate pseudo-ground values, allowing for targeted optimization and dynamic adjustment of key parameters of the Lucas-Kanade algorithm, thereby maximizing the accuracy of the online algorithm while ensuring real-time performance. This collaborative mechanism ensures that the system maintains optimal performance under different road conditions (such as changes in texture richness).

[0090] Based on the foregoing embodiments, this application also discloses the specific content of a preset improved RANSAC algorithm. The step of extracting inliers from the dense optical flow field using the preset improved RANSAC algorithm includes: obtaining the optical flow confidence of each pixel in the dense optical flow field, and determining the spatial weight of each pixel based on the distance between each pixel and the center pixel of the image; wherein the optical flow confidence is determined based on the optical flow vector and structure tensor matrix corresponding to each pixel; determining the sampling probability of each pixel based on the optical flow confidence and the spatial weight, and filtering out target pixels for calculating the vehicle motion model based on the sampling probability; for any target pixel, obtaining the predicted optical flow value of the vehicle motion model for that target pixel, and calculating the optical flow error value between the optical flow vector corresponding to that target pixel and the predicted optical flow value; determining a target inlier judgment threshold that is compatible with that target pixel; if the optical flow error value is less than the target inlier judgment threshold, then that target pixel is determined as an inlier of the vehicle motion model.

[0091] In this embodiment, it is first necessary to obtain the optical flow confidence score of each pixel in the dense optical flow field. The optical flow confidence score is mainly determined based on the optical flow vector and structure tensor matrix corresponding to each pixel. The specific calculation process is as follows: For each pixel, the optical flow is estimated, and the confidence level is calculated: ; in, for The optical flow confidence score (dimensionless) ranges from (0,1], and the larger the value, the more reliable the optical flow estimation. These are the eigenvalues ​​of the structure tensor matrix. ; The ratio of eigenvalues ​​is close to 0.5 when two eigenvalues ​​are similar, indicating a high confidence level. For optical flow constraint error, ; The error threshold is set to 1.0.

[0092] The structure tensor matrix is ​​defined as follows: ; This matrix is ​​the coefficient matrix in Lucas-Kanade optical flow estimation, and its eigenvalues ​​reflect the reliability of the local image structure.

[0093] In this embodiment, the confidence score comprehensively considers the condition number of the structural tensor matrix and the optical flow constraint error. When both eigenvalues ​​are large (e.g., at corner points), the matrix condition is good, and the confidence score is high; when the eigenvalue differences are large or both are small (e.g., at edges or in uniform regions), the condition number is poor, and the confidence score is low. Pixels with low confidence scores will be filtered or have their weight reduced in subsequent processing.

[0094] Furthermore, this application determines the spatial weight of each pixel based on the distance between each pixel and the center pixel of the image. Then, based on the optical flow confidence and spatial weight, the sampling probability of each pixel is determined, thereby prioritizing the sampling of high-confidence pixels in the central region of the image, which are then used to estimate the vehicle motion model. ; in, For the first i The probability of a pixel being selected to participate in the model estimation, all pixels The sum is 1; For the first i Optical flow confidence per pixel Sampling probability The usage is as follows: in each iteration of RANSAC, based on the probability distribution... Three pixels are randomly selected (it is estimated that at least three points are needed for rigid body transformation) to calculate the vehicle motion model parameters. .because With confidence level and spatial weights Proportional, the central region of the image ( Large) and high confidence level of optical flow ( Larger pixels are more likely to be selected, thus prioritizing the sampling of pixels with high reliability and improving the accuracy and efficiency of model estimation.

[0095] Furthermore, for any target pixel, it is necessary to calculate the error between its optical flow and the model prediction. Specifically, the predicted optical flow value of the vehicle motion model for any target pixel is obtained, and the optical flow error value between the optical flow vector corresponding to any target pixel and the predicted optical flow value is calculated. : ; in, For the first i Optical flow error per pixel, Let i be the optical flow vector corresponding to the i-th pixel. To analyze the first vehicle based on the vehicle motion model i The predicted optical flow value for each pixel.

[0096] Then, a threshold for determining the target interior point that matches any target pixel is determined. If the optical flow error value is less than the target interior point judgment threshold, that is... Then any target pixel is determined as an interior point of the vehicle motion model, such as... Figure 3 As shown.

[0097] The step of determining the target interior point judgment threshold that is compatible with any target pixel includes: calculating the second Euclidean distance between any target pixel and the center pixel of the image, and determining the maximum reference distance based on the image width and image height of the bird's-eye view; determining a threshold adjustment factor based on the second Euclidean distance, the maximum reference distance and a preset distance coefficient, and determining the product between the threshold adjustment factor and the preset basic interior point judgment threshold as the target interior point judgment threshold that is compatible with any target pixel.

[0098] In other words, this embodiment adaptively adjusts the intrapoint judgment threshold based on the spatial position of the pixels. A stricter threshold is used for the central region of the image where movement is greater, while a more lenient threshold is used for the edge regions where movement is smaller. ; in, Thresholds are determined for target interior points to match the target pixels. The threshold for determining the in-point basis is set to 2 pixels. The threshold adjustment factor is mainly based on the second Euclidean distance. Maximum reference distance and preset distance coefficient Determined; This represents the Euclidean distance between the target pixel and the center pixel of the image. The value can be 0.5.

[0099] Furthermore, it's important to note that for complex scenes, multiple motion models may exist (such as vehicle motion and dynamic object motion). In such cases, multi-model RANSAC is used: first, the primary motion model (vehicle motion) is estimated; then, secondary motion models (dynamic object motion) are estimated from the remaining points; finally, different models are distinguished based on motion amplitude and spatial distribution. Estimating the primary motion model involves processing the optical flow of all pixels using an improved RANSAC algorithm. Since the vehicle motion affects the largest number of pixels (the entire static background conforms to this motion), RANSAC will find the model with the most inliers as the primary motion model. Remaining points refer to pixels that do not conform to the primary motion model, i.e., pixels judged as outliers in the primary motion model estimation. These pixels typically correspond to dynamic objects in the scene (such as pedestrians or other vehicles), and their motion is inconsistent with the vehicle's own motion. Estimating secondary motion models involves re-running the RANSAC algorithm on the remaining point set to estimate the motion model parameters of the dynamic objects. If the number of remaining points is small or a consistent motion model cannot be formed, it is considered that there are no significant dynamic objects in the scene.

[0100] The distinction between the primary motion model (vehicle's own motion) and the secondary motion model (motion of a dynamic object) is mainly made through three dimensions: The first step is to differentiate the amplitude of motion. Calculate the amplitude of motion for the two motion models. The motion of a vehicle itself usually has a large and stable amplitude, while the amplitude of motion of dynamic objects (such as pedestrians or other vehicles) may be larger or the direction of motion may be different from that of the vehicle.

[0101] Secondly, spatial distribution differentiation is crucial. The interior points of the primary motion model (pixels conforming to the model) should be distributed across the entire image area, because a static background exhibits a consistent optical flow pattern under vehicle motion. In contrast, the interior points of the secondary motion model should cluster in local areas of the image, i.e., the locations of dynamic objects. By statistically analyzing the spatial distribution range of the interior points, it's possible to determine whether the model represents global or local motion.

[0102] Finally, there is the consistency check. The main motion model should be basically consistent with the motion direction and amplitude predicted by the IMU sensor. If a certain model deviates too much from the IMU prediction, then the model is more likely to be the motion of a dynamic object rather than the motion of the vehicle itself.

[0103] Furthermore, the step of estimating global motion parameters based on the set of inliers includes: obtaining initial global motion parameters of the vehicle motion model, constructing a second objective function containing the Huber loss function using the optical flow error values ​​of each point in the set of inliers, and performing nonlinear optimization on the initial global motion parameters using an iterative reweighted least squares method to obtain optimized global motion parameters.

[0104] That is, this embodiment receives vehicle motion model parameters as initial global motion parameters and an interior point set as optimization data, thereby constructing a second objective function containing the Huber loss function using the optical flow error values ​​of each point in the interior point set: ; in, Indicates the motion parameters Optimize to minimize the objective function; The rotation angle to be optimized is initially set to the RANSAC estimated value. , For the x-direction translation to be optimized, the initial value is the RANSAC estimate. , The initial value is the RANSAC estimated value for the y-direction translation to be optimized. ; Let be the set of interior points. For the inner pixel index; For the first i The weight of each in-place point is usually taken from the optical flow confidence score. or spatial weight , Huber loss function; Let be the optical flow error value for i interior points.

[0105] in, Huber's loss function: ; In the formula, Here, e represents the output value of the Huber loss function, and e represents the input error. The threshold value is 3 pixels, used to distinguish between small and large errors. It provides linear loss for large errors, is insensitive to errors, and reduces the impact of outliers.

[0106] Huber loss uses squared loss for precise optimization of small errors (|e|≤3 pixels) and linear loss for large errors (|e|>3 pixels) to reduce the impact of outliers, balancing accuracy and robustness.

[0107] Finally, this embodiment uses the iterative reweighted least squares method to perform nonlinear optimization on the initial global motion parameters to obtain the optimized global motion parameters.

[0108] The optimization is solved using an iterative reweighted least squares method. In each iteration: 1. Calculate the error of each interior point under the current parameters. ; 2. Calculate weights based on Huber loss: ; in, Huber weights are used to reduce the impact of large error points; 3. Update parameters using weighted least squares: ; in, As the confidence weight for optical flow, The weights are Huber weights, and the combined weights are obtained by multiplying the two.

[0109] 4. It converges after 3-5 iterations.

[0110] threshold The selection of pixels is based on the typical noise level of the bird's-eye view. Points with an error of less than 3 pixels are considered reliable inliers and are precisely optimized using squared loss; points with an error of more than 3 pixels may contain outliers and their impact is limited using linear loss.

[0111] As can be seen, since the dense optical flow field provides rich motion information, this application uses an improved RANSAC algorithm for global parameter extraction, improving motion estimation accuracy and reducing translation and rotation errors. Furthermore, this application employs a spatially weighted sampling strategy to prioritize sampling high-confidence pixels at the image center, and adaptive interior point judgment adjusts the threshold based on spatial location, fully utilizing the spatial characteristics of the bird's-eye view, enabling accurate estimation of vehicle motion and effective filtering of dynamic object interference.

[0112] Based on the foregoing embodiments, such as Figure 4 As shown in the embodiments of this application, a process for fusing multi-sensor data using the Kalman filtering method is also disclosed. The process involves fusing the global motion parameters, the first motion parameters measured by the inertial measurement unit, and the second motion parameters measured by the positioning system to obtain fused motion parameters. This includes: constructing a state vector comprising vehicle position, vehicle heading angle, vehicle speed, and vehicle angular velocity; combining the global motion parameters, the first motion parameters measured by the inertial measurement unit, and the second motion parameters measured by the positioning system into a joint observation vector; wherein the global motion parameters include rotation angles and translation vectors, the first motion parameters include vehicle acceleration and vehicle angular velocity, and the second motion parameters include vehicle position and vehicle speed; performing prediction and update steps of extended Kalman filtering to estimate the state vector based on the joint observation vector and the target observation noise covariance matrix to obtain an optimal estimate of the state vector, and using the optimal estimate as the fused motion parameters.

[0113] Understandably, the Kalman filtering process first requires constructing a state vector that includes the vehicle's position, heading angle, velocity, and angular velocity. ; in, For vehicle location, For the vehicle's heading angle, For vehicle speed, This represents the vehicle's angular velocity.

[0114] Furthermore, constructing an observation model specifically requires combining the global motion parameters, the first motion parameters measured by the inertial measurement unit, and the second motion parameters measured by the positioning system into a joint observation vector.

[0115] Among them, the global motion parameters provide the rotation angle changes and translations between adjacent frames, that is, the optical flow observation vector is: ; in, For the change of rotation angle, For translation (pixels), optical flow estimation directly outputs pixel displacement, while maintaining pixel units in Kalman filtering to ensure consistency with the observation function units.

[0116] The optical flow observation model is as follows: ; in, The optical flow observation vector is (3×1). For the optical flow observation function, the state vector x Mapped to the optical flow observation space, Let be the optical flow observation noise vector (3×1), which follows a zero-mean Gaussian distribution with a covariance of . .

[0117] in, ; Angular velocity in the state vector The velocity in the state vector, Δt represents the time interval between adjacent frames. For a 30fps video, Δt = 1 / 30 ≈ 0.033 seconds; s represents the resolution of the bird's-eye view.

[0118] Understandably, the observation function converts the physical velocity (m / s) in the state vector into pixel displacement in the image coordinate system: The unit is meters. Dividing by s (meters / pixel) yields the pixel value, which is then compared with the observation vector. The units must be consistent to ensure that the dimensions of both ends of the Kalman filter observation equation are consistent.

[0119] The first motion parameters measured by the inertial measurement unit are acceleration and angular velocity, i.e., the IMU observation vector is: ; The positioning system can specifically be the GPS (Global Positioning System). GPS observations provide position and velocity (calculated through position differential), i.e., the GPS observation vector is: ; The GPS speed is obtained by the difference between two consecutive GPS position observations: ; ; In the formula, The GPS speed is obtained through differential calculation; The GPS position observation at time t; For the GPS position observation at time t-1; This is the GPS sampling time interval, typically 0.1 seconds (10Hz GPS).

[0120] Velocity estimation is derived from the complementarity of multiple sensors: optical flow provides the velocity of inter-frame displacement conversion (high short-term accuracy), GPS differential provides the global velocity (long-term stability), and IMU acceleration integral provides the high-frequency velocity change (fast response). The three are fused through Kalman filtering and mutually corrected.

[0121] Specifically, the extended Kalman filter fuses data from three sensors using a joint observation vector. The specific fusion mechanism involves stacking the observation vectors from the three sensors into a single joint observation vector. ; in, Let be the joint observation vector (10×1) at time k. The optical flow observation vector (3×1) includes rotation angle changes and translation; The IMU observation vector (3×1) contains acceleration and angular velocity; The GPS observation vector is 4×1, containing position and velocity.

[0122] Furthermore, the prediction and update steps of the extended Kalman filter are performed to estimate the state vector based on the joint observation vector and the target observation noise covariance matrix, so as to obtain the optimal estimate of the state vector, and the optimal estimate is used as the fused motion parameters.

[0123] The target observation noise covariance matrix includes a first noise covariance matrix corresponding to the global motion parameters, a second noise covariance matrix corresponding to the inertial measurement unit, and a third noise covariance matrix corresponding to the positioning system; wherein the first noise covariance matrix is ​​determined based on the average optical flow confidence of the dense optical flow field, the second noise covariance matrix is ​​determined based on the sensor accuracy of the inertial measurement unit, and the third noise covariance matrix is ​​determined based on the positioning accuracy of the positioning system.

[0124] It should also be noted that an observation function is required during the Kalman filtering process. In this embodiment, the corresponding observation function is... It consists of three sub-observation functions: ; Each sub-observation function maps the state vector to the observation space of the corresponding sensor.

[0125] Furthermore, the target observation noise covariance matrix in this application Including the first noise covariance matrix corresponding to the global motion parameters The second noise covariance matrix corresponding to the inertial measurement unit The third noise covariance matrix corresponding to the positioning system .

[0126] Specifically, the target observation noise covariance matrix The noise characteristics of the three sensors are fused into a block-diagonal matrix: ; in, The first noise covariance matrix (3×3) corresponding to the optical flow observation is dynamically adjusted based on the average optical flow confidence of the dense optical flow field. The second noise covariance matrix (3×3) corresponding to the IMU observation is determined by the accuracy of the IMU sensor. The third noise covariance matrix (4×4) corresponding to GPS observations is determined by the GPS positioning accuracy; the block diagonal structure indicates that the noise of each sensor is independent of each other.

[0127] The specific prediction steps are as follows: ; ; in, The state prediction value at time k is a 6×1 vector, based on the state estimate at time k-1. This is the optimal state estimate (6×1 vector) at time k-1. The control input at time k is specifically the acceleration and angular velocity measured by the IMU in this scheme; The state transition function is based on the kinematic model; The state prediction covariance matrix at time k is (6×6). The state estimation covariance matrix at time k-1 is (6×6). for The transpose of the matrix; is the Jacobian of the state transition matrix; Let be the process noise covariance.

[0128] The update steps are as follows: ; ; ; in, The Jacobian of the observation matrix; The target observation noise covariance matrix is ​​fused with the covariance matrices of optical flow, IMU, and GPS. The covariance update adopts the Joseph form to ensure numerical stability and the symmetric positive definiteness of the covariance matrix. The Kalman gain matrix (6×10) determines the update weights of the observation information for the state estimate; This is the optimal state estimate (6×1 vector) after fusing observations at time k. This is the joint observation vector (10×1) at time k. The predicted observations (10×1) are calculated based on the predicted state. To observe the residual, also known as the innovation vector; Let I be the state estimation covariance matrix at time k (6×6); I is the 6×6 identity matrix.

[0129] It should be noted that the Kalman gain The system automatically assigns weights based on the noise covariance matrix of each sensor: sensors with smaller noise covariance (higher accuracy) receive heavier weights, while sensors with larger noise covariance (lower accuracy) receive lighter weights. By updating the formula, the observed and predicted information is optimally fused to obtain a state estimate that integrates information from all sensors.

[0130] Here, the process of determining the first noise covariance matrix based on the average optical flow confidence of the dense optical flow field specifically includes: optimizing the dense optical flow field to obtain an optimized optical flow field, and calculating the average optical flow confidence based on the optimized optical flow field; calculating an initial noise covariance matrix based on the average optical flow confidence and a preset basic noise covariance matrix; and smoothing the initial noise covariance matrix based on a preset smoothing coefficient and the smoothed noise covariance matrix corresponding to historical frames to obtain the currently smoothed first noise covariance matrix.

[0131] It is understandable that this embodiment also requires optimization of the dense optical flow field to obtain an optimized optical flow field, in order to further improve the robustness of optical flow estimation. Based on the optimized optical flow field, the average optical flow confidence is calculated, and then the noise covariance matrix corresponding to the optical flow observation is calculated. It should be noted that traditional extended Kalman filtering uses a fixed observation noise covariance matrix, while this application dynamically adjusts the weights according to the confidence of the optical flow estimation. As can be seen from the foregoing, It is the joint observation noise covariance matrix, composed of block diagonal structures formed by the noise covariance matrices of the three sensors, and... and Using different fixed values It is dynamically adjusted based on the optical flow confidence score: firstly, based on the average optical flow confidence score corresponding to the optimized optical flow field of the current frame. Calculate the initial noise covariance matrix Then, through time smoothing, we obtain... Finally, it was taken as The optical flow submatrix in the Kalman filter is used for updating, thus achieving the goal of dynamically adjusting the fusion weights based on the optical flow estimation quality.

[0132] In a specific implementation, the initial noise covariance matrix of the optical flow observation... Calculated based on average optical flow confidence: ; in, The basic noise covariance matrix, ,in radian, rice; The average optical flow confidence level for all pixels; When the confidence level is high, the noise covariance is small and the optical flow observation weight is large; when the confidence level is low, the noise covariance is large and the optical flow observation weight is small.

[0133] To avoid drastic changes in weights, the initial noise covariance matrix also needs to be smoothed over time. ; in, This is a smoothing coefficient, and its value can be 0.7; This is the smoothed noise covariance matrix corresponding to the previous frame; This is the first noise covariance matrix after smoothing, used for Kalman filter updates. During initialization (first frame), it is used directly. Smoothing coefficient This means retaining 70% of historical information and absorbing 30% of new observations. This approach can respond to actual changes in optical flow quality while avoiding weight abrupt changes caused by single-frame anomalies. This allows for dynamic adjustment of fusion weights based on optical flow estimation quality, making full use of the advantages of each sensor.

[0134] Furthermore, the optimization of the dense optical flow field to obtain the optimized optical flow field includes: filtering the dense optical flow field based on a preset multi-level filtering mechanism to obtain the filtered optical flow field; wherein the multi-level filtering mechanism sequentially includes a first filtering mechanism for maintaining the consistency of the optical flow of each pixel with the optical flow of neighboring pixels, a second filtering mechanism for verifying the difference between the global motion parameters and the motion parameters predicted by the inertial measurement unit, and a third filtering mechanism for smoothing the dense optical flow field; constructing a third objective function containing optical flow constraint terms and smoothing terms based on the filtered optical flow field, and solving the third objective function based on an iterative method to obtain the optimized optical flow field; wherein the optical flow constraint terms are constructed based on the optical flow constraint equation, and the smoothing terms are constructed based on the spatial gradient information of the optical flow.

[0135] That is, such as Figure 5As shown, this application also discloses an outlier filtering and optimization module to improve the robustness of optical flow estimation. The input data for this module includes dense optical flow fields and global motion parameters. and the motion parameters predicted by the inertial measurement unit. The filtering process employs a three-level filtering mechanism, with each level targeting different types of anomalies, as shown in Table 1: Table 1

[0136] The first level is a primary filtering mechanism used to maintain the consistency of optical flow between each pixel and the optical flow of neighboring pixels: Assuming that the motion of adjacent pixels should be similar, if the motion of a pixel differs too much from that of its neighborhood, it may be an outlier.

[0137] For pixels Calculate the difference between its motion and the average motion of its 8-neighborhood: ; in, For pixels The difference in motion between it and its neighboring regions; For pixels The optical flow vector, u is the optical flow in the x direction, v is the optical flow in the y direction; For pixels The set of 8 neighboring pixels, that is, the 8 surrounding adjacent pixels; The coordinates of the pixels in the neighborhood; For neighboring pixels The optical flow vector; Let be the Euclidean norm of the vector.

[0138] if ( If the value is 5 pixels, it is marked as an outlier.

[0139] The second level is a second filtering mechanism used to verify the difference between the global motion parameters and the motion parameters predicted by the inertial measurement unit: If the system has IMU data, use the motion predicted by the IMU to verify the optical flow estimation.

[0140] The rotation angle predicted by the IMU is Translation The difference between the motion parameters estimated by optical flow and those predicted by IMU is calculated: ; ; in, This refers to the rotation angle error; The global rotation angle obtained from optical flow estimation; The rotation angle measured by the IMU sensor; This is the translation error; This is the global translation vector (in meters) obtained from optical flow estimation. ; The translation vector measured by the IMU sensor. ; .

[0141] if ( (Value is 0.05 radians) or ( If the value is 0.2 meters, the weight of the optical flow in that area is reduced, but it is not filtered directly because the IMU may also have errors.

[0142] The third level is a third filtering mechanism used to smooth dense optical flow fields: Spatial smoothness checks were performed on the optical flow field, and median filtering was used to remove isolated outliers. ; in, For pixels Optical flow in the x-direction after median filtering; This is for the median value operation; It is the set of x-direction optical flow values ​​for all pixels within a 5×5 neighborhood (a total of 25 values); In pixels The set of 5×5 neighboring pixels centered on the center; The coordinates of the pixels in the neighborhood; For neighboring pixels The optical flow in the x-direction.

[0143] Similarly, the median filtering of the optical flow in the y-direction is: .

[0144] Furthermore, after filtering outliers, the optical flow field needs to be optimized to improve estimation accuracy and consistency. In this embodiment, a third objective function containing optical flow constraint terms and smoothing terms needs to be constructed based on the filtered optical flow field, and the third objective function is solved based on an iterative method to obtain the optimized optical flow field. The optical flow constraint terms are constructed based on the optical flow constraint equation, and the smoothing terms are constructed based on the spatial gradient information of the optical flow.

[0145] In a specific implementation, a variational method is used to optimize the optical flow field: ; in, Represents the optical flow field Optimize to minimize the objective function; This is the optical flow constraint term, which is ideally zero based on the assumption of constant brightness. The gray-level gradient of the image in the x-direction (gray-level value / pixel); This represents the grayscale gradient (grayscale value / pixel) of the image in the y-direction. The gray-level gradient of the image in the time direction (gray-level value / frame); The optical flow (pixels / frame) in the x and y directions to be optimized; The smoothing weight is set to 0.1, which controls the smoothness of the optical flow field. For the smoothing term, constrain the spatial smoothness of the optical flow field; Let x be the square of the spatial gradient of the optical flow in the x-direction. The square of the spatial gradient of the optical flow in the y-direction.

[0146] This embodiment uses an iterative method to solve the third objective function, updating it in each iteration: ; ; in, This is the optical flow estimate (pixels / frame) after the (k+1)th iteration. This is a smoothed version (pixels / frame) of the optical flow in the k-th iteration, achieved by... Gaussian filtering is applied to obtain the result; k is the iteration number index. ; in molecules The constraint error for the current optical flow estimation; in the denominator As the normalization factor, The terms guarantee numerical stability. The system converges after 5-10 iterations.

[0147] As can be seen, this application fuses motion parameters estimated by optical flow with IMU and GPS data, uses extended Kalman filtering, and dynamically adjusts the fusion weights based on the confidence level of the optical flow estimation. This fully leverages the advantages of each sensor—optical flow, IMU, and GPS—relying more on optical flow when the confidence level is high and relying more on IMU / GPS when the confidence level is low, thus improving the robustness and accuracy of pose estimation. Furthermore, this application designs a three-level filtering mechanism to effectively handle the influence of dynamic objects and changes in illumination, and combines variational optimization to further enhance the robustness of optical flow estimation.

[0148] See Figure 6 As shown in the figure, this application discloses a panoramic imaging device, which includes: Image conversion module 11 is used to perform projection transformation on the original image captured by the vehicle camera to generate a bird's-eye view; The optical flow estimation module 12 is used to determine the size of the target window based on the distance from each pixel in the bird's-eye view to the center pixel of the image, and to perform optical flow estimation on each pixel in two adjacent frames of bird's-eye view based on the vehicle motion information measured by the inertial measurement unit within the target window size, so as to obtain a dense optical flow field. The parameter estimation module 13 is used to extract interior points from the dense optical flow field using a preset improved RANSAC algorithm to obtain an interior point set, and to estimate global motion parameters based on the interior point set. The parameter fusion module 14 is used to fuse the global motion parameters, the first motion parameters measured by the inertial measurement unit, and the second motion parameters measured by the positioning system using the Kalman filtering method to obtain fused motion parameters. The image stitching module 15 is used to calculate the homography matrix based on the fused motion parameters, and to use the homography matrix to project the historical panoramic image onto the bird's-eye view coordinate system. Then, the projected historical panoramic image is stitched together with the current bird's-eye view to generate a new panoramic image.

[0149] As can be seen, this application determines the appropriate target window size based on the distance from each pixel in the bird's-eye view to the image center. Simultaneously, it combines vehicle motion information from the inertial measurement unit (IMU) for optical flow estimation, introducing motion prior constraints. This combination of adaptive windowing and motion priors effectively solves the problem of unstable optical flow estimation in single-texture regions, resulting in a denser optical flow field that better reflects the actual vehicle motion. This application utilizes a pre-defined, improved RANSAC algorithm to extract interior points from the dense optical flow field, effectively filtering outomas caused by dynamic objects and lighting changes, selecting a set of interior points that conforms to the vehicle motion model. The global motion parameters estimated based on this set of interior points accurately reflect the vehicle's rotation and translation states, reducing motion parameter estimation errors and providing reliable foundational data for panoramic stitching. Furthermore, this application fuses the global motion parameters estimated by optical flow, the first motion parameters of the IMU, and the second motion parameters of the positioning system using Kalman filtering, comprehensively utilizing the short-term accuracy advantage of optical flow, the high-frequency response advantage of the IMU, and the long-term stability advantage of the positioning system. Multi-sensor data fusion effectively compensates for the limitations of a single sensor, maintaining high pose estimation accuracy even in complex scenes and avoiding stitching failures due to the malfunction of a single sensor. Finally, based on accurate fused motion parameters, the homography matrix is ​​calculated, enabling accurate projection of the historical panoramic image onto the current bird's-eye view coordinate system. A new panoramic image is then generated by stitching the projected historical panoramic image with the current bird's-eye view. This solution achieves stable, accurate, and efficient panoramic imaging based on optical flow technology.

[0150] For a description of the features in the embodiments corresponding to the panoramic imaging device, please refer to the relevant descriptions in the embodiments corresponding to the aforementioned methods, which will not be repeated here.

[0151] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Specifically, it may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the panoramic imaging method performed by the electronic device disclosed in any of the foregoing embodiments.

[0152] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0153] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0154] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored on it include operating system 221, computer program 222 and data 223, etc., and the storage method can be temporary storage or permanent storage.

[0155] The operating system 221 manages and controls the various hardware devices and computer programs 222 on the electronic device 20 to enable the processor 21 to perform calculations and processing on the massive amounts of data 223 in the memory 22. The operating system 221 can be Windows, Unix, Linux, etc. The computer program 222, in addition to including a computer program capable of performing the panoramic imaging method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, may further include computer programs capable of performing other specific tasks. The data 223 may include data received by the electronic device from external devices, as well as data collected by its own input / output interface 25.

[0156] Furthermore, embodiments of this application also disclose a computer-readable storage medium storing a computer program, which, when loaded and executed by a processor, implements the panoramic imaging method steps disclosed in any of the foregoing embodiments.

[0157] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0158] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0159] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, compact disc read-only memory (CD-ROM), or any other form of storage medium known in the art.

[0160] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0161] The panoramic imaging method, apparatus, device, and storage medium provided by the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A panoramic imaging method, characterized in that, include: The raw images captured by the vehicle's camera are projected and transformed to generate a bird's-eye view; The size of the target window is determined based on the distance from each pixel in the bird's-eye view to the center pixel of the image. Within the target window size, optical flow estimation is performed on each pixel in two adjacent frames of bird's-eye view based on the vehicle motion information measured by the inertial measurement unit to obtain a dense optical flow field. The improved RANSAC algorithm is used to extract interior points from the dense optical flow field to obtain an interior point set, and global motion parameters are estimated based on the interior point set. The global motion parameters, the first motion parameters measured by the inertial measurement unit, and the second motion parameters measured by the positioning system are fused using the Kalman filtering method to obtain fused motion parameters. The homography matrix is ​​calculated based on the fused motion parameters, and the historical panoramic image is projected onto the bird's-eye view coordinate system using the homography matrix. Then, the projected historical panoramic image is stitched together with the current bird's-eye view to generate a new panoramic image.

2. The panoramic imaging method according to claim 1, characterized in that, The step of determining the appropriate target window size based on the distance from each pixel in the bird's-eye view to the center pixel of the image includes: Determine the center pixel of the bird's-eye view image, as well as the image width and height; Calculate the first Euclidean distance between each pixel and the center pixel of the image, and determine the maximum reference distance based on the image width and the image height; The window adjustment factor is determined based on the first Euclidean distance, the maximum reference distance, and the target window adaptive coefficient. The product of the window adjustment factor and the preset base window size is then determined as the target window size that adapts to each pixel.

3. The panoramic imaging method according to claim 2, characterized in that, The step of estimating optical flow for each pixel in two adjacent bird's-eye view frames within the target window size based on vehicle motion information measured by an inertial measurement unit to obtain a dense optical flow field includes: Acquire the vehicle speed and angular velocity measured by the inertial measurement unit; Predict the prior optical flow of each pixel between two adjacent bird's-eye view frames based on the vehicle speed, the angular velocity, the coordinates of the center pixel of the image, and the bird's-eye view resolution. A first objective function containing optical flow constraint terms and prior constraint terms is constructed based on the objective prior weight coefficients; wherein, the optical flow constraint terms are constructed based on the optical flow constraint equation, and the prior constraint terms are constructed based on the prior optical flow; Within the target window size, the first objective function is solved using the weighted least squares method to obtain the optical flow vector of each pixel, and a dense optical flow field is obtained based on each optical flow vector.

4. The panoramic imaging method according to claim 3, characterized in that, Also includes: The current scene type is determined based on the image texture features of the original image; Select target parameter combinations corresponding to the current scene type from a pre-calibrated offline parameter set; the target parameter combinations include the target window adaptive coefficient and the target prior weight coefficient, and the parameter set pre-stores parameter combinations corresponding to different scene types.

5. The panoramic imaging method according to claim 4, characterized in that, The offline calibration process includes: During the offline phase, image sequences covering various scene types are acquired; The optical flow field of the image sequence is calculated using an improved PWC-Net network as a pseudo-true value; wherein, the improved PWC-Net network introduces a spatial attention mechanism in each layer of the feature pyramid, and the total loss function of the network is constructed based on L1 loss, gradient loss and spatial prior loss. With the goal of minimizing the error between the dense optical flow field corresponding to the image sequence and the pseudo-true value, and within a preset parameter search space, the optimal parameter combination is searched for different scene types based on a grid search method to obtain a parameter set.

6. The panoramic imaging method according to claim 1, characterized in that, The extraction of interior points from the dense optical flow field using the pre-defined improved RANSAC algorithm includes: The optical flow confidence of each pixel in the dense optical flow field is obtained, and the spatial weight of each pixel is determined based on the distance between each pixel and the center pixel of the image; wherein, the optical flow confidence is determined based on the optical flow vector and structure tensor matrix corresponding to each pixel; The sampling probability of each pixel is determined based on the optical flow confidence and the spatial weight, and the target pixels for calculating the vehicle motion model are selected based on the sampling probability. For any target pixel, obtain the predicted optical flow value of the vehicle motion model for the target pixel, and calculate the optical flow error value between the optical flow vector corresponding to the target pixel and the predicted optical flow value. A target interior point determination threshold is determined that is compatible with any of the target pixels. If the optical flow error value is less than the target interior point determination threshold, then any of the target pixels is determined as an interior point of the vehicle motion model.

7. The panoramic imaging method according to claim 6, characterized in that, The step of determining the target interior point judgment threshold that matches any of the target pixels includes: Calculate the second Euclidean distance between any target pixel and the center pixel of the image, and determine the maximum reference distance based on the image width and image height of the bird's-eye view; A threshold adjustment factor is determined based on the second Euclidean distance, the maximum reference distance, and a preset distance coefficient. The product of the threshold adjustment factor and the preset basic interior point judgment threshold is determined as the target interior point judgment threshold that is adapted to any target pixel.

8. The panoramic imaging method according to claim 6, characterized in that, The estimation of global motion parameters based on the set of interior points includes: Obtain the initial global motion parameters of the vehicle motion model. A second objective function containing the Huber loss function is constructed using the optical flow error values ​​of each point in the inlier set, and the initial global motion parameters are nonlinearly optimized using iterative reweighted least squares method to obtain the optimized global motion parameters.

9. The panoramic imaging method according to claim 1, characterized in that, The method of fusing the global motion parameters, the first motion parameters measured by the inertial measurement unit, and the second motion parameters measured by the positioning system using Kalman filtering to obtain fused motion parameters includes: Construct a state vector that includes vehicle position, vehicle heading angle, vehicle speed, and vehicle angular velocity; The global motion parameters, the first motion parameters measured by the inertial measurement unit, and the second motion parameters measured by the positioning system are combined into a joint observation vector; wherein, the global motion parameters include rotation angles and translation vectors, the first motion parameters include vehicle acceleration and vehicle angular velocity, and the second motion parameters include vehicle position and vehicle velocity; The extended Kalman filter is executed in the prediction and update steps to estimate the state vector based on the joint observation vector and the target observation noise covariance matrix, so as to obtain the optimal estimate of the state vector, and the optimal estimate is used as the fused motion parameters.

10. The panoramic imaging method according to claim 9, characterized in that, The target observation noise covariance matrix includes a first noise covariance matrix corresponding to the global motion parameters, a second noise covariance matrix corresponding to the inertial measurement unit, and a third noise covariance matrix corresponding to the positioning system; wherein, the first noise covariance matrix is ​​determined based on the average optical flow confidence of the dense optical flow field, the second noise covariance matrix is ​​determined based on the sensor accuracy of the inertial measurement unit, and the third noise covariance matrix is ​​determined based on the positioning accuracy of the positioning system.

11. The panoramic imaging method according to claim 10, characterized in that, The process of determining the first noise covariance matrix based on the average optical flow confidence of the dense optical flow field includes: The dense optical flow field is optimized to obtain an optimized optical flow field, and the average optical flow confidence is calculated based on the optimized optical flow field. Calculate the initial noise covariance matrix based on the average optical flow confidence and the preset basic noise covariance matrix; The initial noise covariance matrix is ​​smoothed based on a preset smoothing coefficient and the smoothed noise covariance matrix corresponding to the historical frames to obtain the first noise covariance matrix after smoothing.

12. The panoramic imaging method according to claim 11, characterized in that, The optimization of the dense optical flow field to obtain the optimized optical flow field includes: The dense optical flow field is filtered based on a preset multi-level filtering mechanism to obtain a filtered optical flow field; wherein, the multi-level filtering mechanism includes, in sequence, a first filtering mechanism for maintaining the consistency of the optical flow of each pixel with the optical flow of neighboring pixels, a second filtering mechanism for verifying the difference between the global motion parameters and the motion parameters predicted by the inertial measurement unit, and a third filtering mechanism for smoothing the dense optical flow field. Based on the filtered optical flow field, a third objective function containing optical flow constraint terms and a smoothing term is constructed, and the third objective function is solved using an iterative method to obtain the optimized optical flow field; wherein, the optical flow constraint term is constructed based on the optical flow constraint equation, and the smoothing term is constructed based on the spatial gradient information of the optical flow.

13. The panoramic imaging method according to any one of claims 1 to 12, characterized in that, The step of stitching the projected historical panoramic image with the current bird's-eye view to generate a new panoramic image includes: Determine the overlapping area between the projected historical panoramic view and the current bird's-eye view; The fusion weight coefficient is calculated based on the difference in acquisition timestamps and the average value of the optical flow confidence of each pixel in the overlapping region; wherein, the difference in acquisition timestamps is the difference between the acquisition timestamp of the current bird's-eye view and the acquisition timestamp corresponding to the overlapping region in the historical panoramic image; The projected historical panoramic image and the current bird's-eye view are weighted and fused based on the fusion weight coefficient to obtain a fused image; The fused image is processed using multi-band fusion technology to eliminate seams, resulting in a new panoramic image.

14. A panoramic imaging device, characterized in that, include: The image conversion module is used to project and transform the raw images captured by the vehicle camera to generate a bird's-eye view; The optical flow estimation module is used to determine the size of the target window based on the distance from each pixel in the bird's-eye view to the center pixel of the image, and to perform optical flow estimation on each pixel in two adjacent frames of bird's-eye view based on the vehicle motion information measured by the inertial measurement unit within the target window size, so as to obtain a dense optical flow field. The parameter estimation module is used to extract interior points from the dense optical flow field using a preset improved RANSAC algorithm to obtain an interior point set, and to estimate global motion parameters based on the interior point set. The parameter fusion module is used to fuse the global motion parameters, the first motion parameters measured by the inertial measurement unit, and the second motion parameters measured by the positioning system using the Kalman filtering method to obtain fused motion parameters. The image stitching module is used to calculate the homography matrix based on the fused motion parameters, and to use the homography matrix to project the historical panoramic image onto the bird's-eye view coordinate system. Then, the projected historical panoramic image is stitched together with the current bird's-eye view to generate a new panoramic image.

15. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the panoramic imaging method as described in any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that, Used to store a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the panoramic imaging method as described in any one of claims 1 to 13.

Images