A Method and System for Generating Vehicle Panoramic Surround View Based on Image Stitching

Abstract

This invention discloses a method and system for generating panoramic vehicle surround view based on image stitching, belonging to the fields of computer vision and intelligent driving assistance technology. The method includes: a multi-camera synchronous acquisition step, which simultaneously acquires images of the vehicle's surrounding environment through four wide-angle cameras; an adaptive distortion correction step, which performs distortion correction processing on each image based on a distortion correction lookup table; a virtual viewpoint projection step, which projects each image onto a virtual top-down plane with the vehicle's geometric center as the origin; an optical flow motion detection step, which identifies dynamic targets in overlapping areas and generates a dynamic mask; a multi-band fusion step based on image gradients, which uses gradient-weighted fusion for static areas and a single-source priority strategy for dynamic areas; and an adaptive exposure equalization step, which dynamically adjusts the fusion weights according to brightness differences. This invention solves the problems of inaccurate distortion correction, obvious stitching seams, ghosting of dynamic objects, and uneven exposure in existing technologies, significantly improving the quality of panoramic images.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and intelligent driving assistance technology, specifically relating to a method and system for generating a panoramic view of a vehicle based on image stitching, used to provide the driver with a panoramic view of the vehicle's surrounding environment. Background Technology

[0002] With the continuous growth of car ownership and the increasing complexity of road traffic environments, the problem of blind spots faced by drivers during reversing, parking, and low-speed driving is becoming increasingly prominent. As an important component of advanced driver assistance systems, the vehicle surround view system provides drivers with a 360-degree overhead view of the vehicle's surroundings by installing multiple wide-angle cameras around the vehicle and stitching them together, effectively eliminating blind spots and significantly improving driving safety.

[0003] Chinese patent application CN103617606A discloses a method for generating multi-view panoramic images of vehicles for assisted driving. This method involves installing several wide-angle cameras around the vehicle to capture images, correcting distortion and performing perspective transformation on the images by calibrating the intrinsic and extrinsic parameters of the cameras, determining the fusion boundary of adjacent ground projection images to obtain a bird's-eye view, building a 3D model, and projecting near-field and far-field images onto different regions. Finally, a lookup table is generated through multi-view observation to output the panoramic image. While this method can preserve scene information above the horizon and solve the problem of far-field scene stretching, it still has the following technical shortcomings: First, the method uses a fixed 3D model for projection transformation, and the projection parameters remain unchanged after calibration, making it unable to adapt to brightness changes under different lighting conditions. Second, the method only uses a simple boundary fusion strategy in the image fusion stage, which easily produces obvious brightness discontinuities when there are exposure differences between adjacent camera images. Third, the method does not consider dynamic scene processing; when pedestrians or vehicles cross overlapping areas, motion ghosting occurs, affecting the driver's accurate judgment of the surrounding environment.

[0004] To address the aforementioned technical issues, those skilled in the art have attempted to introduce multi-band fusion technology to improve the stitching seam problem. However, traditional multi-band fusion methods use fixed fusion weights, which cannot adaptively adjust according to image content, resulting in significant differences in fusion effects between textured and flat regions. Furthermore, existing dynamic target detection methods often employ frame differencing or background modeling methods, which are computationally complex and sensitive to camera shake, making real-time processing difficult on automotive embedded platforms.

[0005] Therefore, there is an urgent need for a vehicle panoramic surround view generation method that can simultaneously solve the problems of distortion correction accuracy, stitching seam fusion quality, dynamic target ghosting elimination, and exposure imbalance, in order to meet the growing demand for driving safety assistance and users' higher expectations for image quality. Summary of the Invention

[0006] To address the technical problems of existing vehicle panoramic surround view systems, such as inaccurate distortion correction, obvious stitching seams, ghosting of dynamic objects, and uneven exposure, this invention provides a vehicle panoramic surround view generation method and system based on image stitching. By integrating adaptive distortion correction, virtual viewpoint projection, optical flow motion detection, multi-band fusion based on image gradient, and adaptive exposure equalization, high-quality vehicle panoramic surround view images are generated.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A vehicle panoramic surround view generation method based on image stitching includes the following steps: a multi-camera synchronous acquisition step, in which four wide-angle cameras installed at the front, rear, left, and right of the vehicle simultaneously acquire images of the vehicle's surrounding environment, wherein the field of view of each wide-angle camera is not less than 180 degrees, and the fields of view of adjacent cameras overlap; an adaptive distortion correction step, in which a distortion correction lookup table is established based on the pre-calibrated intrinsic parameter matrix and distortion coefficients of each camera, and distortion correction processing is performed on the acquired images to obtain a distortion-free image; and a virtual viewpoint projection step, in which a virtual top-view plane with the vehicle's geometric center as the origin is established based on the vehicle's 3D model, and perspective distortion is calculated based on the extrinsic parameter matrix of each camera. The process involves several steps: First, the distortion-free images are projected onto a virtual top-view plane according to perspective transformation, resulting in top-view projected images. Second, optical flow motion detection identifies dynamic target regions in overlapping areas of adjacent top-view projected images, generating dynamic target masks. Third, multi-band fusion based on image gradients eliminates seams and brightness discontinuities in static areas, while prioritizing single-source images for dynamic target regions. Fourth, adaptive exposure equalization calculates the average brightness of each image and dynamically adjusts the fusion weights based on the average brightness differences, outputting the final panoramic vehicle image.

[0009] A vehicle panoramic surround view generation system based on image stitching, the system includes:

[0010] The multi-camera synchronous acquisition module is configured to simultaneously acquire images of the vehicle's surrounding environment through four wide-angle cameras installed at the front, rear, left, and right of the vehicle body. The field of view of each wide-angle camera is not less than 180 degrees, and there is an overlapping area in the field of view of two adjacent cameras.

[0011] The adaptive distortion correction module is configured to establish a distortion correction lookup table based on the pre-calibrated intrinsic parameter matrix and distortion coefficients of each camera, and perform distortion correction processing on the acquired images to obtain a distortion-free image.

[0012] The virtual viewpoint projection module is configured to establish a virtual top-view plane with the vehicle's geometric center as the origin based on the vehicle's 3D model, calculate the perspective transformation relationship according to the extrinsic matrix of each camera, and project the distortion-free images of each path onto the virtual top-view plane.

[0013] The optical flow motion detection module is configured to identify dynamic target regions and generate dynamic target masks by using optical flow motion detection methods for the overlapping areas of two adjacent top-view projection images.

[0014] The multi-band fusion module is configured to use a multi-band fusion strategy based on image gradient for static regions and a single-source image priority strategy for dynamic target regions.

[0015] The adaptive exposure equalization module is configured to dynamically adjust the fusion weights based on the average brightness of each image path, and output the final panoramic surround view image of the vehicle.

[0016] The beneficial effects of this invention are as follows: A unified coordinate system centered on the vehicle is established through a virtual viewpoint projection algorithm, resulting in less geometric deformation and approximately 30% reduction in image distortion at edge regions compared to existing 3D model projection methods; dynamic targets are identified through optical flow motion detection using a single-source priority strategy, effectively eliminating motion ghosting and achieving a dynamic target recognition accuracy of over 95%; adaptive texture fusion is achieved through a multi-band fusion strategy based on image gradients, significantly improving the visual effect of stitching seams and reducing the visibility of stitching boundaries by approximately 50%; and smooth transitions between bright and dark scenes are achieved through adaptive exposure equalization, significantly improving overall image quality consistency and increasing the uniformity of transitions in areas of brightness difference by approximately 40%. The six processing steps of this invention form a deeply coupled closed-loop collaborative architecture, with the output of each step serving as the key input for the next, achieving end-to-end high-quality panoramic image generation. Attached Figure Description

[0017] Figure 1 This is a flowchart of the vehicle panoramic surround view generation method based on image stitching according to the present invention.

[0018] Figure 2 This is an architecture diagram of the vehicle panoramic surround view generation system based on image stitching of the present invention. Detailed Implementation

[0019] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0020] See Figure 1As shown, the vehicle panoramic surround view generation method based on image stitching provided by the present invention includes six core steps, and each step forms a deeply coupled closed-loop processing flow. The output of the previous step serves as the key input of the next step, and the result of the subsequent step can inversely affect the parameter adjustment of the previous step, thereby achieving collaborative optimization.

[0021] Step S1: Simultaneous data acquisition from multiple cameras.

[0022] In one embodiment of the present invention, the multi-camera synchronous acquisition step is the data input foundation for the entire panoramic surround view generation process. The system uses four wide-angle cameras, respectively installed in the center of the front grille, below the left rearview mirror, below the right rearview mirror, and above the rear license plate. This layout ensures that the field of view of the four cameras completely covers the 360-degree range around the vehicle, while there is sufficient overlap area between adjacent cameras for subsequent image stitching and fusion. The front camera is installed at a height of approximately 0.6m to 0.8m above the ground, with its optical axis tilted downward at approximately 45 degrees, and is mainly responsible for acquiring images of the front of the vehicle and the left and right front diagonal directions. The rear camera is installed at a height of approximately 0.5m to 0.7m above the ground, with its optical axis also tilted downward at approximately 40 degrees, and is mainly responsible for acquiring images of the rear of the vehicle and the left and right rear diagonal directions. The left and right side cameras are installed below the rearview mirrors, approximately 1.0m to 1.2m above the ground, with their optical axes tilted downward at approximately 30 degrees, and are responsible for acquiring images of the left and right sides of the vehicle, respectively. This asymmetrical installation layout and tilt angle configuration fully consider the vehicle's geometric shape and the need to focus on the ground area in typical driving scenarios, effectively eliminating blind spots present in traditional single-camera or dual-camera systems.

[0023] Preferably, each camera uses a fisheye lens to obtain a wider field of view. In this embodiment, a wide-angle camera with a field of view of 190 degrees is selected, with a horizontal field of view of 170 degrees and a vertical field of view of 120 degrees. The selection of the fisheye lens needs to comprehensively consider image quality, distortion characteristics, and cost factors. This embodiment recommends using an equidistant projection fisheye lens, as the distortion curve of this type of lens is relatively smooth and approximately linear, which is beneficial to the numerical stability of subsequent distortion correction processing. The field of view overlap area of ​​two adjacent cameras is approximately 45 degrees. This overlap angle has been optimized to ensure that there are enough feature points for registration during stitching and fusion without causing excessive image data redundancy that would affect real-time processing performance. The image resolution of the cameras is configured as 1920×1080 pixels, the frame rate is set to 30fps, and YUV422 format is used for output to balance image quality and data transmission bandwidth. The image sensor uses a 1 / 2.8-inch CMOS chip with a pixel size of 2.9μm and a dynamic range of no less than 120dB, ensuring that clear images can be acquired under both strong direct light and low light shadow conditions, meeting the driving assistance needs in complex lighting environments.

[0024] In another embodiment of the present invention, to ensure the time synchronization of images acquired by the four cameras, the system adopts a hardware-triggered synchronization mechanism. Specifically, the main controller sends trigger signals to the four cameras simultaneously at a period of 33.33ms. Upon receiving the trigger signal, each camera synchronously starts exposure and acquires one frame of image. The synchronization error is controlled within 1ms. For a typical parking scenario with a vehicle speed below 30km / h, the image misalignment caused by the synchronization error does not exceed 1cm, fully meeting the accuracy requirements of panoramic image stitching. During the system initialization phase, the exposure parameters of each camera also need to be pre-configured, including setting the upper limit of gain for automatic exposure mode to 16dB and the upper limit of exposure time to 30ms, to ensure that each camera can acquire images with similar brightness under different lighting conditions.

[0025] Step S2: Adaptive distortion correction.

[0026] Due to the inherent optical characteristics of fisheye lenses, the acquired raw images exhibit significant barrel distortion, with the distortion being particularly severe in the image edge regions. In one embodiment of the present invention, the adaptive distortion correction step first obtains the intrinsic parameter matrix and distortion coefficients of each camera through offline calibration, and then establishes a distortion correction lookup table to support real-time image correction processing.

[0027] The camera calibration phase uses a checkerboard calibration board. In this embodiment, a 10×7 checkerboard calibration board with 30mm side length per square is selected. During the calibration process, the calibration board is placed at different positions and angles within the camera's field of view, and 20 to 30 calibration images with different orientations are captured. For fisheye lenses, this invention uses an equidistant projection model for distortion modeling. This model can more accurately describe the imaging characteristics of large field-of-view lenses compared to pinhole camera models.

[0028] Specifically, let the intrinsic parameter matrix of the fisheye lens be... The intrinsic parameter matrix of the k-th camera can be represented as: ,in: and These are the focal lengths of the k-th camera in the x and y directions, respectively, in pixels, ranging from 200 to 500 pixels. The specific values ​​are determined by the camera sensor size and lens focal length. and , respectively, represent the position of the optical center of the k-th camera in the image coordinate system. Ideally, it should be located at the center of the image. The deviation between the actual calibration value and the theoretical center is usually within 10 pixels. The subscript k represents the camera number. k takes values ​​of 1, 2, 3, and 4, corresponding to the front, left, right, and rear cameras, respectively.

[0029] Distortion coefficient vector of fisheye lens A four-parameter model is used to describe: ,in: , , , is the radial distortion coefficient, used to describe the degree of distortion in the radial direction of the image. Its value typically ranges from -0.5 to 0.5, with negative values ​​indicating barrel distortion and positive values ​​indicating pincushion distortion; the superscript T indicates vector transpose. For the 190-degree field-of-view fisheye lens used in this embodiment, a typical distortion coefficient is... , , , .

[0030] In another embodiment of the present invention, to achieve real-time distortion correction, the system pre-calculates a distortion correction lookup table during the initialization phase. Let the original image coordinates be... The corrected image coordinates are Lookup table The storage mapping relationship is as follows: ,in: This is a distortion correction lookup table for the k-th camera. The table size is consistent with the image resolution, which is 1920×1080 in this embodiment. The pixel coordinates in the distorted image represent the index position of the lookup table; The corresponding pixel coordinates in the original distorted image are stored as floating-point numbers to support sub-pixel precision interpolation.

[0031] The calculation of the lookup table involves a reverse mapping, that is, for each pixel position in the distorted image, calculating its corresponding original image coordinates. First, the distorted image coordinates are converted into normalized camera coordinates: ,in: and The normalized camera coordinates are represented by coordinate values ​​in units of focal length, and are dimensionless; other symbols have the same meaning as before. Then, the original image coordinates are calculated based on the isometric projection model and distortion coefficients. The specific algorithm uses an iterative solution method, and convergence is usually achieved in 5 to 10 iterations.

[0032] During real-time processing, distortion correction is performed through a lookup table operation. For each output pixel, the corresponding original image coordinates are read from the lookup table, and bilinear interpolation is used to obtain the pixel value. ,in: For the distortion-free image of the k-th camera, in coordinates Pixel value at; For the original image in coordinates The pixel value at that location is a floating-point coordinate given by a lookup table. Determine the neighborhood location; The bilinear interpolation weights are calculated from the fractional part of the sub-pixel coordinates and satisfy the following conditions: By using a lookup table and bilinear interpolation, the distortion correction processing time for a single frame of four images is controlled within 5ms, meeting real-time requirements.

[0033] Step S3: Virtual viewpoint projection.

[0034] In one embodiment of the present invention, the core of the virtual viewpoint projection step is to establish a virtual top-view plane with the vehicle's geometric center as the origin, and to project the various distortion-free images onto this plane through perspective transformation to form a top-view image in a unified coordinate system. Unlike the prior art that uses a three-dimensional bowl-shaped or spherical model, the present invention uses a virtual top-view plane, which can avoid geometric distortion caused by the curved surface of the three-dimensional model, while reducing computational complexity.

[0035] Establishing a virtual top-down view requires first defining a world coordinate system. In this embodiment, the projection of the vehicle's geometric center onto the ground is used as the origin, with the x-axis pointing to the right side of the vehicle, the y-axis pointing forward, and the z-axis pointing vertically upward, forming a right-handed coordinate system. The virtual top-down view is located 2.0m above the ground, i.e., a horizontal plane with z=2.0m. This height setting allows the top-down image to completely cover an approximately 10m × 10m ground area around the vehicle, while avoiding obstruction from the vehicle roof that could affect image integrity. For vehicles of different sizes, the height of the virtual top-down view can be adjusted between 1.5m and 3.0m. A height of 2.5m is recommended for taller vehicles such as SUVs, while a height of 1.8m is suitable for shorter vehicles such as sedans.

[0036] Preferably, the image resolution of the top-view plane is set to 2cm per pixel, meaning a 500×500 pixel top-view image corresponds to a 10m×10m actual area. This resolution setting clearly displays key features in parking scenarios such as ground markings, curbs, and parking space lines, while maintaining reasonable computational complexity and display quality. For applications requiring higher precision, the resolution can be increased to 1cm per pixel, with the corresponding top-view image size increasing to 1000×1000 pixels. The vehicle's 3D model is built at a 1:1 scale, with its center coinciding with the origin of the world coordinate system. The vehicle model uses a simplified bounding box format, including key geometric features such as the vehicle body outline and wheel positions, which are used to overlay the vehicle outline in the final panoramic image, providing the driver with an intuitive spatial distance reference.

[0037] In another embodiment of the present invention, the calculation of the perspective transformation matrix requires the extrinsic parameter matrices of each camera, including rotation matrices and translation vectors. Let the extrinsic parameter of the k-th camera be the rotation matrix. Translation vector ,in A 3×3 orthogonal matrix is ​​used to describe the rotation relationship between the camera coordinate system and the world coordinate system. The 3×1 vector describes the position of the camera's optical center in the world coordinate system.

[0038] Extrinsic parameter calibration is performed by arranging calibration feature points on the ground. In this embodiment, 16 calibration feature points are placed on the ground around the vehicle, forming a 4×4 grid with a grid spacing of 1m. The calibration feature points are high-contrast circular markers with a diameter of 10cm, facilitating image detection and precise positioning. Each camera acquires images containing visible calibration points. The image coordinates of the calibration points are detected through image processing, and combined with the known world coordinates, the extrinsic parameter matrix is ​​solved using the PnP algorithm.

[0039] Perspective transformation matrix A mapping relationship between the coordinates of the distorted image and the coordinates of the top view plane was established. Let the pixel coordinates in the distorted image be... The corresponding top-view plane coordinates are Then we have: ,in: The scale factor for homogeneous coordinates is an inherent parameter of perspective transformation and is dimensionless. Let be the perspective transformation matrix of the k-th camera, which is a 3×3 non-singular matrix, calculated jointly from the intrinsic and extrinsic parameter matrices; The coordinates of a point on the top-view plane in the world coordinate system are given in meters (m), with the z-coordinate fixed at 2.0m.

[0040] Perspective transformation matrix The calculation formula is: ,in: , , Rotation matrices The first, second, and third column vectors; The height is taken as 2.0m for the top-view plane. This represents the column concatenation operation of the matrix. The perspective transformation matrix can be directly calculated from the calibrated intrinsic and extrinsic parameters using the above formula, without the need for additional calibration.

[0041] To achieve real-time processing, this invention also uses a lookup table to store the perspective transformation mapping relationship. Let the coordinates of the top view image be... Lookup table storage: ,in: This is a perspective transformation lookup table for the k-th camera. These are the pixel coordinates in the top-view projection image, and they are the indexes of the lookup table; This provides the corresponding pixel coordinates in the distorted image. The lookup table is calculated and stored during system initialization. In real-time processing, image projection is completed through table lookup and bilinear interpolation. The perspective transformation processing time for the four images is controlled within 3ms.

[0042] Step S4: Optical flow motion detection.

[0043] In one embodiment of the present invention, the optical flow motion detection step identifies dynamic targets in the overlapping region of the top-view projection image by analyzing the motion information of pixels between adjacent frames. When pedestrians or other vehicles cross the overlapping region, the position and posture of the dynamic target differ in the two images due to the different shooting angles of the two adjacent cameras. Direct image fusion would result in obvious ghosting. The present invention introduces optical flow motion detection technology, which can effectively identify and mark dynamic target regions, providing a basis for subsequent adoption of a single-source priority strategy.

[0044] The overlapping area is determined based on the effective mapping range of perspective transformation. For four cameras (front, rear, left, and right), the overlapping areas of two adjacent cameras are located in the four diagonal directions of the vehicle, namely the front left, front right, rear left, and rear right. Let the range of the overlapping area between adjacent cameras k-th and k+1-th cameras on the top-view plane be... Each pixel in this region exists simultaneously in both overhead projection images.

[0045] In another embodiment of the present invention, the optical flow calculation employs the pyramid Lucas-Kanade sparse optical flow method, which achieves a good balance between computational efficiency and motion detection accuracy. First, feature points are detected on the top-view projection image. This embodiment uses the Shi-Tomasi corner detection algorithm to detect 100 to 200 feature points in each overlapping region. Then, the optical flow vector of the feature points between two adjacent frames is calculated. The number of pyramid layers is set to 4, the number of iterations is set to 20, and the convergence threshold is set to 0.03 pixels.

[0046] Let the coordinates of the feature points on the t-th frame be... The coordinates of the corresponding point in the (t+1)th frame are Then the optical flow vector Defined as: ,in: This is an optical flow vector, representing the displacement of a feature point between two adjacent frames, in pixels per frame; and These are the components of the optical flow vector in the x and y directions, respectively. The magnitude of the optical flow vector... It reflects the target's speed of movement.

[0047] In one embodiment of the present invention, the discrimination of dynamic targets is based on an optical flow amplitude threshold. Let the motion threshold be... Feature points with optical flow amplitudes greater than a threshold are marked as dynamic points.

[0048] ,in: This is the dynamic label for feature point p, with a value of 1 indicating a dynamic point and a value of 0 indicating a static point; The motion threshold is measured in pixels per frame. In this embodiment, it is set to 3 pixels per frame. This threshold corresponds to an actual movement speed of approximately 0.2 times the vehicle speed, which can effectively distinguish between dynamic targets and errors caused by camera shake.

[0049] Preferably, to generate a continuous dynamic target mask from sparse dynamic feature points, this invention employs a morphological processing method. First, a circular region with radius r, centered on each dynamic feature point and set to 20 pixels, is created. Then, all circular regions are merged to obtain an initial dynamic region. Next, closing and opening operations are performed sequentially to fill holes and remove noise, with the morphological kernel size set to 15×15 pixels. Finally, the boundaries of the dynamic region are smoothed to obtain the final dynamic target mask. .

[0050] Furthermore, this invention introduces a temporal filtering mechanism to improve the robustness of dynamic detection. Let the dynamic mask of the current frame be... The dynamic mask for the first N frames is Then the filtered dynamic mask is:

[0051] ,in: This is a dynamic target mask after time-series filtering; The time-series filtering window size is set to 5 frames in this embodiment; The activation threshold is set to 3, meaning that a region is only considered a dynamic region if at least 3 out of the most recent 6 frames detect motion. This temporal filtering mechanism effectively suppresses false detections caused by transient noise while maintaining a sensitive response to continuously moving targets.

[0052] Step S5: Multi-band fusion based on image gradient.

[0053] In one embodiment of the present invention, the multi-band fusion step based on image gradients employs a frequency-division processing strategy to achieve seamless image fusion for the static portions of overlapping regions. Traditional multi-band fusion methods use a fixed distance weight function, applying the same fusion strategy to texture-rich and flat regions, which easily leads to texture blurring or edge ghosting. The present invention introduces image gradient information to guide the fusion weight allocation, enabling adaptive adjustment of the fusion strategy based on local image features, significantly improving the stitching effect.

[0054] The basic idea of ​​multi-band fusion is to decompose an image into sub-band components of different frequencies, fuse each sub-band separately, and then reconstruct the complete image. This invention uses a Laplacian pyramid to achieve frequency decomposition, with 5 pyramid layers. Let the original image be... Then the various levels of the Gaussian pyramid The calculation formula is: , ,in: This is an image of the l-th layer of the Gaussian pyramid, where l is counted starting from 0. This is the original image; The smoothing kernel is Gaussian; in this embodiment, a 5×5 Gaussian kernel with a standard deviation of 1.0 is used. This represents the convolution operation; This indicates a 2x downsampling operation, using an alternating row and column sampling method.

[0055] Each level of the Pyramid of Laplace The following can be obtained by subtracting adjacent layers of the Gaussian pyramid: ,in: This is an image of the l-th layer of the Laplacian pyramid, containing high-frequency detail information of the original image at this scale; This indicates a 2x upsampling operation, implemented using bilinear interpolation. The highest level of the Laplace pyramid is the same as the highest level of the Gaussian pyramid. Where L is the total number of pyramid layers minus 1, and in this embodiment L=4.

[0056] In another embodiment of the present invention, the fusion weights based on image gradients are calculated as follows. First, the gradient magnitudes of each top-view projection image are calculated: ,in: For the k-th path image in coordinates The gradient magnitude at a given location, expressed in grayscale values ​​per pixel, reflects the richness of texture at that location. and These are the partial derivatives of the image in the x and y directions, respectively, calculated using the Sobel operator.

[0057] In overlapping areas Within, let two adjacent image paths be respectively and Then the gradient-based fusion weights The calculation is as follows: ,in: For the k-th path image in coordinates The fusion weight at the point has a value range of [0, 1]. The gradient sensitivity index is set to 2 in this embodiment. This parameter controls the sensitivity of the fusion weights to gradient differences. To prevent small constants from being divided by zero, set to Accordingly, the weight of the (k+1)th path image is... .

[0058] Preferably, to ensure a smooth transition at the fusion boundary, this invention superimposes a distance weight on top of the gradient weight. (Setting a point...) The distance to the center line of the overlapping region is The half-width of the overlapping region is Then distance weight Using a cosine transition function: ,in: As a distance weight, it takes a value of 0.5 at the center line of the overlapping area and smoothly transitions to 0 or 1 at the boundary; The sign of the point is determined by its orientation relative to the center line, and its range is [value range missing]. ; The width of the overlapping area is approximately 25 pixels in this embodiment.

[0059] Final fusion weights Combining gradient weights and distance weights: ,in: For comprehensive integration and weighting; The gradient weight adjustment coefficient is set to 0.3 in this embodiment. This parameter controls the degree of influence of the image gradient on the fusion weight. If the value is too large, it may cause discontinuities in the boundary; if the value is too small, the adaptive fusion effect will be lost. The fusion weight is truncated to ensure that the value is within the range of [0, 1].

[0060] Weighted fusion is performed on each level of the Laplace pyramid: ,in: This is the first layer of the merged Laplace's Pyramid; and These are the l-th layer of the Laplacian pyramid for the two images respectively; The fusion weight mask for layer l is composed of the comprehensive fusion weights. The result is obtained through downsampling. It should be noted that low-frequency layers (with a larger l) mainly use distance weights to achieve a smooth transition, while high-frequency layers (with a smaller l) rely more on gradient weights to preserve texture details.

[0061] Finally, the merged Laplace pyramid is reconstructed to obtain the final merged image:

[0062] .

[0063] In another embodiment of the present invention, a single-source image priority strategy is used instead of multi-band fusion for dynamic target regions marked by optical flow detection. Let the dynamic target mask be... The fusion strategy within the dynamic region is then: ,in: The preferred source image is selected based on the principle that the moving target in the image is closer to the center of the camera's field of view and has better image quality. Specifically, the centroid position of the moving target region is calculated, and then the distance from the centroid to the center of the field of view of both cameras is calculated. The image from the camera with the closer distance is selected as the preferred source.

[0064] To avoid abrupt changes at the boundary between dynamic and static regions, this invention employs a feathering transition at the boundary of the dynamic mask. Let the feathering width be... Then the boundary transition weight is:

[0065] ,in: This is the transition weight, with a value range of [0, 1]. For point Distance to the boundary of the dynamic mask, in pixels; The feather width is set to 20 pixels in this embodiment. Within the feathered area, the final fusion result is: ,in: This is the result of multi-band fusion. Through this feathering transition mechanism, a smooth transition is achieved between dynamic and static regions, avoiding visible boundary traces.

[0066] Step S6: Adaptive Exposure Equalization.

[0067] In one embodiment of the present invention, the adaptive exposure equalization step compensates for possible brightness differences between images from different cameras, ensuring a natural transition in the panoramic image at the boundary between light and shadow. In actual driving scenarios, lighting conditions often vary significantly in different directions of the vehicle. For example, one side of the vehicle may be directly illuminated by the sun while the other side is in shadow. Without exposure equalization, the fused panoramic image will exhibit obvious brightness blockages, affecting the driver's accurate perception of the surrounding environment.

[0068] Brightness statistics are performed in the non-overlapping regions of each top-view projection image to avoid interference from the fusion results of overlapping regions in the brightness calculation. Let the non-overlapping region of the k-th image be... The average brightness of the image on that path The calculation is as follows: ,in: The average brightness of the k-th image is [0, 255], corresponding to an 8-bit grayscale representation. This represents the total number of pixels in the non-overlapping region. The value represents the luminance component of an image pixel. For color images, the Y component of the YUV color space is used, and the calculation formula is as follows: , where R, G, and B are the values ​​of the red, green, and blue channels, respectively.

[0069] In another embodiment of the present invention, the target brightness Take the arithmetic mean of the average brightness of the four images: ,in: The target brightness serves as a reference for exposure equalization. This averaging strategy balances brightness information from all directions, preventing over-adjustment in other directions caused by using a single direction as a reference.

[0070] Exposure balance coefficient Calculated based on the ratio of the brightness of each image channel to the brightness of the target: ,in: is the exposure equalization coefficient for the k-th image, with a value range limited to [0.5, 2.0] to avoid over-adjustment leading to image quality degradation; To prevent small constants from being divided by zero, set it to 1. When This indicates that the image is too dark and needs to be brightened. This indicates that the image is too bright and needs to be darkened.

[0071] Preferably, to avoid overexposed or underexposed areas in the image after exposure equalization, this invention employs a non-linear exposure adjustment function. Let the original pixel value be... The adjusted pixel value ,but:

[0072] ,in: The pixel values ​​after exposure adjustment are kept within the range of [0, 255]. The gamma correction factor is related to the exposure equalization factor as follows: This nonlinear relationship can suppress extreme changes in highlight and shadow areas while maintaining overall brightness adjustment.

[0073] During the fusion process in overlapping regions, the exposure equalization coefficient acts as an additional weighting factor, influencing the fusion result. Let the fusion weight be... Therefore, the overall weighting for considering exposure balance is: ,in: The final fusion weights take into account three factors: geometric location, image gradient, and exposure balance. In this way, brighter images receive higher weights in their dominant regions, while less bright images receive lower weights, thus achieving a smooth transition in brightness at the fusion boundary.

[0074] In one embodiment of the present invention, the final panoramic image generation requires combining the fused and non-fused regions of the four images. Let the panoramic image be... ,but:

[0075] ,in: This is the top-view projection image of the k-th path; This is the union of all overlapping regions. The single-path image with adjusted exposure is used directly in the non-overlapping regions, while the fused result is used in the overlapping regions to obtain a complete panoramic view of the vehicle.

[0076] Furthermore, this invention overlays a top-down outline of the vehicle's 3D model in the center of the panoramic image. The vehicle outline is displayed in a semi-transparent green with a transparency set to 0.5, providing the driver with an intuitive reference for the vehicle's position and orientation. The entire panoramic image generation process is completed within 30ms, meeting the real-time display requirement of 30fps.

[0077] See Figure 2 As shown, the present invention also provides a vehicle panoramic surround view generation system based on image stitching. The system includes six functional modules, each of which corresponds to one of the steps in the method embodiment, and they are connected by data stream to form a complete processing pipeline.

[0078] The multi-camera synchronous acquisition module 1 includes four wide-angle cameras and their corresponding synchronous control unit. As described in the method embodiment, the four cameras are respectively installed at the front, rear, left, and right positions of the vehicle body, with a field of view of 190 degrees, an image resolution of 1920×1080 pixels, and a frame rate of 30fps. The synchronous control unit is implemented using an FPGA and sends synchronous trigger signals to the four cameras through a GPIO interface, with a synchronization accuracy better than 1ms. The image data acquired by the cameras is transmitted to the image processing main chip through a MIPI CSI-2 interface, with a single-channel interface bandwidth of 1.5Gbps, meeting the requirements for real-time image transmission.

[0079] The adaptive distortion correction module 2 is responsible for distortion correction of the acquired images. During system initialization, this module loads pre-calibrated camera intrinsic parameters and distortion coefficients from non-volatile memory, calculates and caches the distortion correction lookup table. In real-time processing, this module completes image correction through table lookup and bilinear interpolation, using GPU parallel computing for acceleration. The distortion correction processing time for four images is controlled within 5ms.

[0080] Virtual viewpoint projection module 3 implements perspective transformation from the distorted image to the top-view plane. This module also uses a lookup table to achieve real-time projection, and the perspective transformation lookup table is calculated and generated based on the camera's extrinsic parameters during initialization. As described in the method embodiment, the virtual top-view plane is located 2.0m above the ground with the vehicle's geometric center as the origin, and the resolution is 2cm of actual distance per pixel.

[0081] The optical flow motion detection module 4 employs the Lucas-Kanade sparse optical flow algorithm to identify dynamic targets. This module detects feature points within overlapping regions and tracks their inter-frame motion. Dynamic points are identified by an optical flow amplitude threshold, which, as described in the method embodiment, is set to 3 pixels per frame. The dynamic target mask is generated through morphological processing and temporal filtering, with a filtering window of 5 frames.

[0082] The multi-band fusion module 5 employs gradient-based Laplacian pyramid fusion for static regions and a single-source priority strategy for dynamic regions. As described in the method embodiment, the pyramid has 5 layers, the fusion weights comprehensively consider image gradient and distance factors, and the boundaries of dynamic regions are feathered with a width of 20 pixels.

[0083] The adaptive exposure equalization module 6 dynamically adjusts the fusion weights based on the brightness differences of each image path. As described in the method embodiment, brightness statistics are performed in non-overlapping areas, the exposure equalization coefficient is limited to the range of [0.5, 2.0], and gamma correction is used to achieve non-linear brightness adjustment, ultimately outputting an exposure-equalized panoramic image.

[0084] The above six modules are deployed on an in-vehicle embedded platform. This embodiment uses the NVIDIA Jetson AGX Xavier as the main processing chip, which integrates a 512-core Volta GPU and an 8-core ARM CPU, achieving a computing power of 32 TOPS, meeting the requirements for real-time image processing. The system software is developed based on the Linux operating system, and the image processing algorithms are implemented using CUDA and OpenCV libraries. Distortion correction and perspective transformation are accelerated using texture memory, multi-band fusion is optimized using shared memory, and optical flow calculation is implemented in parallel using CUDA cores. The storage subsystem is configured with 2GB of dedicated video memory for image buffering and lookup table storage, ensuring high-bandwidth image data access. The processing latency of the entire system is controlled within 40ms, with image acquisition and transmission approximately 5ms, distortion correction approximately 5ms, perspective transformation approximately 3ms, optical flow detection approximately 8ms, multi-band fusion approximately 12ms, exposure equalization approximately 4ms, and image display approximately 3ms. The output frame rate is consistently above 25fps, and the panoramic image is displayed on the in-vehicle central control screen with a resolution of 1280×720 pixels or higher and a refresh rate of no less than 60Hz, providing the driver with a clear and distortion-free panoramic view of the vehicle's surroundings. The system also supports video recording, with panoramic images compressed and stored in H.264 format at a bitrate of 8Mbps, which can be used for post-accident scene playback and analysis.

[0085] Compared with existing technologies, the embodiments of this invention have the following technical advantages: By establishing a unified coordinate system centered on the vehicle through virtual viewpoint projection, geometric distortion is reduced by approximately 30% compared to 3D bowl-shaped model projection; through optical flow motion detection and a single-source priority strategy, the success rate of dynamic target ghosting elimination reaches over 95%; through gradient-based multi-band fusion, the visibility of stitching seams is reduced by approximately 50%; and through adaptive exposure equalization, the brightness uniformity of transition areas between bright and dark scenes is improved by approximately 40%. These technical improvements result in panoramic surround view images generated by this invention having significantly better quality than existing solutions, providing drivers with more accurate and reliable information about their surrounding environment.

[0086] The embodiments of the present invention are not limited to the specific embodiments described above. Those skilled in the art can make various equivalent changes or substitutions based on the technical solutions of the present invention, and all such changes or substitutions should be included within the protection scope of the present invention.

Claims

1. A method for generating a vehicle panoramic surround view based on image stitching, characterized in that, The method includes the following steps: S1. Multi-camera synchronous acquisition steps: Simultaneously acquire images of the vehicle's surrounding environment using four wide-angle cameras installed at the front, rear, left, and right of the vehicle. The field of view of each wide-angle camera is not less than 180 degrees, and there is an overlapping area between the fields of view of two adjacent cameras. S2. Adaptive distortion correction steps: Based on the pre-calibrated intrinsic parameter matrix and distortion coefficients of each camera, a distortion correction lookup table is established, and distortion correction processing is performed on the acquired images to obtain the distortion-free image; S3. Virtual viewpoint projection steps: Based on the vehicle's 3D model, establish a virtual top-view plane with the vehicle's geometric center as the origin. Calculate the perspective transformation relationship according to the extrinsic parameter matrix of each camera. Project each distortion-free image onto the virtual top-view plane according to the perspective transformation relationship to obtain each top-view projection image. S4. Optical Flow Motion Detection Step: For the overlapping area of ​​two adjacent top-view projection images, an optical flow motion detection method is used to identify dynamic target regions and generate a dynamic target mask. The dynamic target regions include the motion regions generated when pedestrians and vehicles cross the overlapping area. The optical flow motion detection method includes: calculating the sparse optical flow field between two adjacent top-view projection images. The optical flow calculation adopts the pyramid Lucas-Kanade method with 3 to 5 pyramid layers; performing amplitude and direction clustering analysis on the optical flow vector, and marking regions with optical flow amplitudes greater than a preset motion threshold as dynamic candidate regions. The motion threshold is 2 to 5 pixels per frame; performing morphological processing on the dynamic candidate regions, including dilation and erosion operations, to generate a connected dynamic target mask. S5. Multi-band fusion step based on image gradient: For static areas in the overlapping region, a multi-band fusion strategy based on image gradient is used for image fusion to eliminate stitching seams and brightness discontinuities. The multi-band fusion strategy based on image gradient includes: constructing a Laplacian pyramid for each top-view projection image, with 4 to 6 layers; calculating the image gradient magnitude of each pixel in the overlapping region, generating a fusion weight mask based on the gradient magnitude, with regions having higher fusion weights for larger gradient magnitudes; performing weighted fusion at each layer of the Laplacian pyramid, with a gradual transition weight for low-frequency layers and a gradient-based weight for high-frequency layers. The weights are adjusted; the fused Laplacian pyramid is reconstructed to obtain a seamlessly fused panoramic image; for the dynamic target region marked by the dynamic target mask, a single-source image priority strategy is used instead of fusion processing to avoid motion ghosting; the single-source image priority strategy includes: determining the distance between the dynamic target and the center of each camera's field of view based on the position of the dynamic target in the overlapping region; selecting the camera image closer to the center of the field of view as the priority source image; within the dynamic target mask region, the priority source image is directly used to replace the fusion result, and a feathering transition is used at the mask boundary, with a feathering width of 10 to 30 pixels; S6. Adaptive Exposure Equalization Step: Calculate the average brightness of each image, dynamically adjust the fusion weights based on the difference in average brightness, perform exposure equalization processing on the fused panoramic image, so that the panoramic image transitions naturally in scenes where light and dark meet, and output the final panoramic view image of the vehicle.

2. The method according to claim 1, characterized in that, In step S1, four wide-angle cameras are respectively installed in the center of the front grille of the vehicle, below the left and right rearview mirrors, and above the rear license plate. The field of view of each camera is 180 degrees to 220 degrees, the overlap area of ​​the field of view of adjacent cameras is 30 degrees to 60 degrees, the resolution of the acquired image is 1280×720 pixels to 1920×1080 pixels, and the frame rate is 25fps to 30fps.

3. The method according to claim 1, characterized in that, In step S2, the camera calibration uses a checkerboard calibration board with a checkerboard size of 8×6 to 12×9 corner points and a single grid side length of 20mm to 50mm. The resolution of the distortion correction lookup table is consistent with the resolution of the acquired image, and the lookup table stores the coordinate mapping relationship between each pixel of the distorted image and the original image.

4. The method according to claim 1, characterized in that, In step S3, the virtual top-view plane is located 1.5m to 3.0m above the ground. The resolution of the top-view plane is 1cm to 5cm per pixel corresponding to the actual ground. The scale of the vehicle 3D model is 1:

1. The perspective transformation matrix is ​​obtained by calibrating the correspondence between ground feature points and image pixels.

5. The method according to claim 1, characterized in that, In step S6, the adaptive exposure equalization step includes: calculating the average brightness value of each top-view projection image in the non-overlapping area; using the arithmetic mean of the average brightness of the four images as the target brightness; calculating the exposure equalization coefficient based on the ratio of the average brightness of each image to the target brightness; and using the exposure equalization coefficient as an additional weighting factor when merging overlapping areas to achieve a smooth brightness transition.

6. The method according to claim 5, characterized in that, The method also includes a lookup table pre-calculation step: during the system initialization phase, a distortion correction lookup table and a perspective transformation lookup table are pre-calculated and stored. During the real-time processing phase, image transformation is completed through lookup table operations to achieve real-time panoramic image generation with a processing frame rate of no less than 25fps.

7. A vehicle panoramic surround view generation system based on image stitching, used to implement the method described in any one of claims 1-6, characterized in that, The system includes: The multi-camera synchronous acquisition module is configured to simultaneously acquire images of the vehicle's surrounding environment through four wide-angle cameras installed at the front, rear, left, and right of the vehicle body. The field of view of each wide-angle camera is not less than 180 degrees, and there is an overlapping area in the field of view of two adjacent cameras. The adaptive distortion correction module is configured to establish a distortion correction lookup table based on the pre-calibrated intrinsic parameter matrix and distortion coefficients of each camera, and perform distortion correction processing on the acquired images to obtain a distortion-free image. The virtual viewpoint projection module is configured to establish a virtual top-view plane with the vehicle's geometric center as the origin based on the vehicle's 3D model, calculate the perspective transformation relationship according to the extrinsic matrix of each camera, and project the distortion-free images of each path onto the virtual top-view plane. The optical flow motion detection module is configured to identify dynamic target regions and generate dynamic target masks by using optical flow motion detection methods for the overlapping areas of two adjacent top-view projection images. The multi-band fusion module is configured to use a multi-band fusion strategy based on image gradient for static regions and a single-source image priority strategy for dynamic target regions. The adaptive exposure equalization module is configured to dynamically adjust the fusion weights based on the average brightness of each image path, and output the final panoramic surround view image of the vehicle.

Images