Surrounding view stitching method and driving system based on geometric lie group feature extraction and manifold optimization

Abstract

The present application belongs to the field of computer vision and intelligent driving technology, and discloses a kind of geometric Lie group feature extraction and manifold optimization's all-around view splicing method and driving system, including using the multi-channel image acquisition equipment of ring array composition to collect multi-channel perspective image;For each perspective image, the regional geometric feature descriptor is constructed using the regional geometric feature description method based on Lie group structure modeling;Based on the geometric feature descriptor corresponding to the perspective image, the manifold splicing optimization method based on matrix Lie group structure is used to complete multi-channel perspective image fusion;The overall image after fusion and the overlapping area of adjacent two perspective images are fused and optimized to obtain a preliminary splicing image;The quality of the preliminary splicing image is evaluated, and if the quality evaluation fails, the fusion is performed again;Otherwise, output the preliminary splicing image as the all-around view panoramic image obtained by all-around view splicing.The present application aims to realize high-precision and high-efficiency panoramic image splicing.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and intelligent driving, and particularly relates to a surround view stitching method based on geometric Lie group feature extraction and manifold optimization and a driving system. BACKGROUND

[0002] With the rapid development of intelligent driving technology and the continuous improvement of automatic driving level, the demand for panoramic view of the vehicle environment perception system becomes increasingly urgent. The monocular or binocular camera scheme adopted by the traditional vehicle-mounted vision system has inherent defects such as limited viewing angle range and obvious visual blind area. To effectively solve this technical problem, the 360-degree surround view system based on multi-camera image stitching technology has gradually developed into the core perception scheme in the current intelligent driving field. Among them, the phase correlation method based on frequency domain transformation has advantages in calculation speed, but its adaptability to image rotation and scale change is poor. The image stitching algorithm based on regional image descriptor has excellent scale invariance and rotation invariance characteristics, but the clustering method for dense features will cause a lot of information loss, thereby affecting the performance of the descriptor.

[0003] More importantly, in the actual road running environment, the variable light conditions such as the rapid change of light and shade at the entrance and exit of the tunnel, the mottled light under the tree shade, etc. will seriously affect the stability of feature point detection. Actual test data shows that the fluctuation range of feature point matching accuracy under such environmental conditions can reach 15%. At the same time, the image occlusion caused by dynamic obstacles on the road such as adjacent vehicles, pedestrians, etc. will cause about 18% of the stitching failure rate. In addition, the images collected by the multi-camera system during vehicle movement also have the problem of inter-frame difference, which causes obvious motion blur and ghosting phenomenon. In the image fusion link, the traditional average value fusion method will produce obvious visual seams in the stitching boundary area, which seriously affects the visual presentation effect of the system. These factors together constitute the technical bottleneck that needs to be broken through in the current intelligent driving environment perception field, and it is urgent to develop a solution that can stably run in complex road environments while meeting the requirements of real-time, high precision and strong robustness. SUMMARY

[0004] To solve the problems of insufficient real-time, low matching accuracy and poor environmental adaptability of the image stitching technology in the existing surround view system, the present application provides a surround view stitching method based on geometric Lie group feature extraction and manifold optimization and a driving system, which is suitable for environment perception of unmanned vehicles and aims to realize high-precision and high-efficiency panoramic image stitching.

[0005] To achieve the above-mentioned purpose, the technical solution adopted by the present application is as follows:

[0006] In a first aspect, a surround view stitching method based on geometric Lie group feature extraction and manifold optimization is provided, comprising:

[0007] a plurality of view images are collected by a plurality of image collection devices arranged in a ring array;

[0008] for each view image, a regional geometric feature descriptor is constructed by a regional geometric feature description method based on Lie group structure modeling;

[0009] based on the geometric feature descriptors corresponding to the view images, a manifold stitching optimization method based on matrix Lie group structure is used to complete the fusion of the plurality of view images;

[0010] the overall fused image and the overlapping region between two adjacent view images are optimized to obtain a preliminary stitched image;

[0011] the quality of the preliminary stitched image is evaluated, if the quality evaluation fails, the fusion is re-performed, otherwise the preliminary stitched image is output as a ring panoramic image obtained by ring view stitching.

[0012] The following also provides several optional modes, but not as an additional limitation to the above overall scheme, just a further supplement or preferred, without technical or logical contradiction, each optional mode can be combined with the above overall scheme, and can also be combined between multiple optional modes.

[0013] As a preferred, the plurality of view images are collected by a plurality of image collection devices arranged in a ring array, comprising:

[0014] a plurality of image collection devices are arranged in a ring array, adjacent two image collection devices have overlapping view angles, and each image collection device is calibrated;

[0015] a uniform clock signal source is used to synchronously trigger the plurality of image collection devices to collect raw images;

[0016] the raw images are converted to a vehicle coordinate system according to a rigid transformation relationship from a camera coordinate system to the vehicle coordinate system, and the converted images are adaptively white balanced to obtain corrected images;

[0017] nonlinear gamma correction processing is introduced to perform nonlinear mapping transformation on the brightness of the corrected images to obtain view images.

[0018] As a preferred, the regional geometric feature descriptor is constructed by a regional geometric feature description method based on Lie group structure modeling, comprising:

[0019] the view image is blurred and down-sampled by a five-layer Gaussian pyramid structure, and each layer of the Gaussian pyramid generates a Gaussian blurred image, a number of sub-layers of the five-layer Gaussian pyramid structure;

[0020] a plurality of Gaussian blur images are selected from each Gaussian pyramid, and a feature matrix of the selected Gaussian blur image is extracted;

[0021] a Type II multivariate Laplace distribution parameter estimation is performed on each feature matrix of each Gaussian blur image, and the estimated Type II multivariate Laplace distribution parameter is embedded into a Lie group structure to obtain an affine embedding matrix, and a first type of feature descriptor is obtained by performing a logarithmic mapping on the affine embedding matrix;

[0022] a polar decomposition is performed on the affine embedding matrix by introducing a Lie group quotient structure optimization strategy to obtain a left polar positive definite matrix and a right polar positive definite matrix, and a second type of feature descriptor and a third type of feature descriptor are obtained by performing a logarithmic mapping on the left polar positive definite matrix and the right polar positive definite matrix, respectively.

[0023] As a preferred, the geometric feature descriptor corresponding to the perspective image is fused by a manifold splicing optimization method based on a matrix Lie group structure, comprising:

[0024] The geometric feature descriptors of adjacent perspective images are matched, and the RANSAC method is used to eliminate the false matches to obtain a matching pixel point set;

[0025] The Lie group is used to perform a global alignment optimization on the adjacent perspective images based on the matching pixel point set to obtain a global homography matrix;

[0026] The target image corresponding to the global homography matrix is divided into a plurality of grid sub-regions, and a local homography matrix of each grid sub-region is estimated;

[0027] A global similarity transformation is extracted from the global homography matrix, and the local homography matrix and the global similarity transformation are interpolated on the Lie group to obtain an interpolation transformation matrix of each grid sub-region;

[0028] The fusion of adjacent perspective images is realized by fusing a plurality of interpolation transformation matrices between adjacent perspective images, and the fusion of each adjacent two perspective images is performed to finally output a fused image, thereby completing the fusion of multiple perspective images.

[0029] As a preferred, the fusion optimization of the entire fused image and the overlapping region of adjacent two perspective images is performed to obtain a preliminary spliced image, comprising:

[0030] The adjacent perspective images are aligned to the same coordinate system according to the difference transformation matrix between the adjacent perspective images, and the dynamic fusion weight between each pair of matching pixel points in the overlapping region of the adjacent perspective images is calculated, and the pixel value of the pixel point in the overlapping region of the fused image is updated based on the dynamic fusion weight.

[0031] The updated fused image is corrected by using a multi-scale Gaussian pyramid;

[0032] The corrected fused image is subjected to adaptive histogram equalization processing and color mapping correction based on a matching region to obtain a preliminary stitching image.

[0033] Preferably, the contrast enhancement intensity in the adaptive histogram equalization processing is adjusted by using a PID controller, and an error signal of the PID controller is defined as a difference between a preset structural similarity index of the preliminary stitching image and an actual structural similarity index of the preliminary stitching image.

[0034] Preferably, the preliminary stitching image is subjected to quality assessment, and if the quality assessment fails, the fusion is re-performed; otherwise, the preliminary stitching image is output as a final surround view stitching panoramic image, including:

[0035] If the root mean square error of the continuous multiple frames of preliminary stitching images is greater than a fifth threshold value, the quality assessment fails, and the step of obtaining the set of matching pixel points is re-performed;

[0036] If the structural similarity index of the preliminary stitching image is less than a second threshold value and a difference between the current structural similarity index and a previous structural similarity index is greater than a decline threshold value, the quality assessment fails, and the step of correcting the updated fused image by using a multi-scale Gaussian pyramid is re-performed;

[0037] Otherwise, it is indicated that the quality assessment passes, and the preliminary stitching image is output as a final surround view stitching panoramic image.

[0038] Preferably, the surround view stitching method based on geometric Lie group feature extraction and manifold optimization further includes dynamic parameter adaptive adjustment, specifically:

[0039] If the structural similarity index of the preliminary stitching image is less than a first threshold value, the contrast suppression coefficient in the adaptive histogram equalization processing is enhanced;

[0040] If the root mean square error of the preliminary stitching image is greater than a third threshold value, the inlier judgment threshold value of the RANSAC method is reduced;

[0041] If the correct matching rate of the preliminary stitching image is less than a fourth threshold value, the confidence threshold value of the RANSAC method is increased.

[0042] In a second aspect, a driving system is provided, including: a multi-path image acquisition device, an image data processing and analysis device, a data information recording device, a vehicle control unit, a data communication device, and a terminal device, wherein:

[0043] The multi-channel image acquisition device is used to acquire original images with overlapping perspectives;

[0044] The image data processing and analysis device is used to perform the steps of the look-around stitching method for implementing the geometric Lie group feature extraction and manifold optimization;

[0045] The data communication device is used to realize data transmission between the multi-channel image acquisition device and the image data processing and analysis device;

[0046] The data recording device is used to record the data involved in the geometric Lie group feature extraction and manifold optimization look-around stitching method;

[0047] The vehicle control unit is used to control the vehicle by combining the most panoramic view image;

[0048] The terminal device is used to display panoramic images.

[0049] The surround-view stitching method and driving system for geometric Lie group feature extraction and manifold optimization provided by this invention have the following advantages compared with the prior art:

[0050] The 360-degree panoramic stitching method provided in this invention constructs a more stable and structure-aware feature representation method during image registration based on region-level geometric feature modeling and matrix Lie group structure optimization. Through multivariate Laplace distribution modeling and Lie group embedding, it performs deep geometric description of the multidimensional features of local image regions, effectively preserving the algebraic and geometric relationships of multi-scale features. Transformation optimization and interpolation fusion performed on the manifold enhance deformation control and stitching continuity during image registration. The proposed method achieves high-precision feature matching in complex lighting, occlusion, and parallax scenarios, and generates structurally stable and boundary-smooth panoramic stitching results by constructing a highly consistent local homography matrix and interpolation transformation function. Simultaneously, the integration of a multi-scale pyramid image fusion strategy and a color mapping correction mechanism significantly improves the visual naturalness of the stitched images. By constructing a closed-loop control system driven by quality evaluation indicators such as the structural similarity index (SSIM), root mean square error (RMSE), and correct matching rate (CMR), real-time adaptive adjustment of parameters in the stitching process is achieved, ensuring excellent robustness and stability of the system under varying environments. Attached Figure Description

[0051] Figure 1 The flowchart shows the view stitching method for geometric Lie group feature extraction and manifold optimization of the present invention.

[0052] Figure 2 This is a schematic diagram showing the installation positions of the six 2-megapixel wide-angle cameras of this invention;

[0053] Figure 3 This is a schematic diagram of the driving system of the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention.

[0056] Example 1:

[0057] like Figure 1 As shown, this embodiment provides a look-around stitching method for geometric Lie group feature extraction and manifold optimization. This method uses a five-layer Gaussian pyramid feature extraction and Lie group... Manifold optimization and multi-scale dynamic fusion achieve high-precision image stitching in complex driving environments, and a closed-loop quality assessment is employed to ensure system robustness. This method can achieve panoramic image stitching in vehicular environments with varying illumination and dynamic occlusion, offering better geometric consistency and real-time performance compared to traditional methods. The specific steps include:

[0058] Step 1: Use a multi-channel image acquisition device composed of a ring array to acquire multi-view images.

[0059] Step 1.1: Use multiple image acquisition devices to form a ring array, with overlapping viewing angles between adjacent image acquisition devices, and perform internal parameter calibration on each image acquisition device.

[0060] This embodiment uses a circular array of six 2-megapixel wide-angle cameras, with each camera having a horizontal field of view of [missing information]. The vertical field of view is Installation location such as Figure 2 As shown. The camera spacing has been optimized to ensure overlap in the fields of view of adjacent cameras. This provides structural support for subsequent regional Laplace feature extraction and cross-view matching.

[0061] The intrinsic parameters of each camera were calibrated using Zhang Zhengyou's calibration method to construct a standard pinhole camera imaging model. Pixel coordinates 3D coordinates in the camera coordinate system The following relationship is satisfied between them:

[0062] (1)

[0063] wherein, and is a pixel focal length, is a principal point coordinate.

[0064] The camera intrinsic matrix and distortion coefficients are obtained by Zhang Zhengyou calibration method :

[0065] (2)

[0066] (3)

[0067] wherein, , and are radial distortion coefficients, and are tangential distortion coefficients. The calibration process uses a 12x9 checkerboard and collects 50 groups of images with different poses.

[0068] Step 1.2, the original image is collected by synchronously triggering the multi-channel image acquisition device with a unified clock signal source. In this embodiment, a hardware synchronous trigger is used to realize accurate synchronization of multiple cameras: the trigger signal has a precision of ; the inter-frame time deviation ; supports external event triggering mode. The synchronization system communicates with the vehicle-mounted ECU (Electronic Control Unit) through the CAN (Controller Area Network) bus and monitors the working state of each camera in real time.

[0069] Step 1.3, according to the rigid transformation relationship from the camera coordinate system to the vehicle coordinate system, the original image is converted to the vehicle coordinate system, and the converted image is corrected by adaptive white balance to obtain a corrected image.

[0070] To realize the spatial positioning of the image in the vehicle-mounted system, the rigid transformation relationship from the camera coordinate system to the vehicle coordinate system is established:

[0071] (4)

[0072] wherein, is a rotation matrix, is a translation vector, is a three-dimensional position in the vehicle coordinate system, ​​​This represents the three-dimensional position in the camera coordinate system.

[0073] To improve image color consistency and cross-view fusion stability, adaptive white balance correction is performed on each frame. This is achieved through analysis... Based on the average color temperature of the three channels, the following gain compensation transformation matrix is ​​established:

[0074] (5)

[0075] in, express Three-channel average value , , Each represents the average value of each channel individually. For input Three-channel pixel values For output Three-channel pixel values. This step forms a complementary closed loop with the color mapping in subsequent step 4.3.

[0076] Step 1.4: Introduce nonlinear gamma correction processing to perform nonlinear mapping transformation on the brightness of the corrected image to obtain the viewpoint image.

[0077] To improve detail in low-light areas while suppressing highlight overexposure, this invention introduces nonlinear gamma correction processing. The gamma coefficient is set... A nonlinear mapping transformation is performed on the image brightness:

[0078] (6)

[0079] in, This is the output image after nonlinear gamma correction processing. The input image undergoes nonlinear gamma correction processing. This preprocessing enhances the image's dynamic range, providing more stable grayscale and texture information support for subsequent regional geometric feature extraction.

[0080] Step 2: For each viewpoint image, construct a geometric feature descriptor using a region geometric feature description method based on Lie group structure modeling.

[0081] This embodiment, based on the multi-view images acquired in step 1, employs a region geometric feature description method based on Lie group structure modeling to replace the traditional SIFT feature extraction method. This method enhances the stability and robustness of the image region under complex lighting, occlusion, and other interference conditions while ensuring rotation and scale invariance.

[0082] Step 2.1: Use a five-layer Gaussian pyramid structure to blur and downsample the viewpoint image, and generate each layer of the Gaussian pyramid. Gaussian blurred images, The number of sub-layers is five.

[0083] The original image is blurred and down-sampled using a five-layer Gaussian pyramid structure to ensure the stability and coverage of subsequent regional feature extraction in multiple scales. The input original image , the reference scale , the number of sub-layers , and a five-layer Gaussian pyramid is constructed in an exponential relationship. Each scale space is blurred by a standard Gaussian kernel and generates Gaussian blurred images, and the scale parameter is calculated according to the formula:

[0084] (7)

[0085] The initial image of the first layer is obtained by 2 times down-sampling of the third image of the first layer. The final output is a 5-layer x 6-image scale space graph with a maximum scale . This step provides multi-scale feature map support for geometric feature descriptor modeling.

[0086] Step 2.2, select multiple Gaussian blurred images in each layer of the Gaussian pyramid, and extract the feature matrix of the selected Gaussian blurred images.

[0087] In the five-layer Gaussian pyramid, three sub-layer images (scale parameters are 1.6, 3.2 and 6.4) are selected in each layer to extract pixel regions , from which dimensional feature vectors including brightness, normalized components, chrominance components, edge gradients, and texture information are extracted:

[0088] (8)

[0089] wherein, is the brightness (1 dimension), is the normalized RGB component (3 dimensions), is the YCbCr chrominance component (2 dimensions), is the robust edge gradient (2 dimensions), is the texture information (10 dimensions). Each pixel region calculates dimensional feature vectors to form a feature matrix , .

[0090] Step 2.3, Type II multivariate Laplace distribution parameter estimation is performed on the feature matrix of each Gaussian blur image, and the estimated Type II multivariate Laplace distribution parameters are embedded into the Lie group structure to obtain an affine embedding matrix, and a first type of feature descriptor is obtained by performing logarithmic mapping on the affine embedding matrix.

[0091] The feature matrix is subjected to Type II multivariate Laplace distribution parameter estimation:

[0092] (9)

[0093] wherein, is a mean vector, is a scale parameter matrix. This distribution can more flexibly model the skewness and heavy tail characteristics in image data, and is suitable for feature expression of image regions in actual complex scenes.

[0094] To improve the structural expression capability and matching stability of the distribution modeling, the Laplace distribution parameters and are embedded into the Lie group structure. Through Cholesky decomposition , an affine embedding matrix is constructed:

[0095] (10)

[0096] wherein, is an affine embedding matrix, is a lower triangular matrix, and a first type of feature descriptor is obtained by performing logarithmic mapping on the affine embedding matrix, which has good illumination robustness and scale adaptability:

[0097] (11)

[0098] Step 2.4, the Lie group quotient structure optimization strategy is introduced to perform polar decomposition on the affine embedding matrix to obtain a left polar positive definite matrix and a right polar positive definite matrix, and logarithmic mapping is performed on the left polar positive definite matrix and the right polar positive definite matrix to obtain a second type of feature descriptor and a third type of feature descriptor.

[0099] To further improve the structural expression capability, the Lie group quotient structure optimization strategy is introduced. By performing polar decomposition on the affine embedding matrix , a left polar positive definite matrix and a right polar positive definite matrix are obtained:

[0100] (12)

[0101] (13)

[0102] The second and third types of feature descriptors are obtained by logarithmic mapping of left and right positive definite matrices, respectively, and are expanded into symmetric Lie algebra space vectors:

[0103] (14)

[0104] For each feature matrix Steps 2.3 and 2.4 are performed to obtain the corresponding geometric feature descriptors, and feature descriptors of the same type are combined, i.e., each type of feature descriptor contains 15 descriptors of the same type corresponding to 15 pixel regions. Then the feature descriptors are further expanded into structure-preserving vectors with a dimension of for subsequent matching and registration steps. This method preserves the algebraic structure and geometric consistency within the image region, and has higher description accuracy and recognition performance in complex lighting and dynamic occlusion environments. Finally, all feature descriptors are output in vector form, which are the original Lie algebra vector corresponding to the first type of feature descriptor , the left extreme vector corresponding to the second type of feature descriptor , and the right extreme vector corresponding to the third type of feature descriptor .

[0105] Step 3, based on the geometric feature descriptors corresponding to the view images, a manifold stitching optimization method based on matrix Lie group structure is used to complete multi-view image fusion. This embodiment introduces an image stitching optimization method based on matrix Lie group structure to realize high-precision alignment and low-distortion fusion between images. This process combines the Lie group manifold structure of spatial transformation, embeds homomorphism transformation into or matrix manifold, and improves the naturalness and accuracy of stitching through a geometric error-driven optimization process.

[0106] Step 3.1, based on the geometric feature descriptors of adjacent view images, feature matching is performed on adjacent view images, and RANSAC method is used to remove false matches to obtain a set of matching pixel points.

[0107] For adjacent view image pairs and , based on the geometric feature descriptors of step 2, a double screening mechanism based on Euclidean distance and cosine similarity is used to complete regional-level feature matching:

[0108] (15)

[0109] The matching pairs of the pixel points with the matching confidence descriptors of the three types of feature descriptors higher than the confidence threshold are reserved, and then a homography matrix is fitted by a RANSAC method , abnormal matching points not meeting the geometric constraint are removed, and finally a high-precision matching pixel point set is output , is the pixel point in the view image (the pixel point coordinate in the reference image), is the pixel point in the view image (the pixel point coordinate in the target image corresponding to ), is the total number of pixel point matching pairs, used for subsequent transformation optimization, is the geometric feature descriptor of the view image , is the geometric feature descriptor of the view image . Step 3.2, based on the matching pixel point set, the global alignment optimization on the Lie group between adjacent view images is carried out to obtain a global homography matrix.

[0110] Special Linear Group (SLG) is a kind of Lie group, which is composed of all 3x3 invertible matrices with determinant 1. The global homography matrix between image pairs is regarded as an element in the Lie group, and is mapped to the Lie algebra

[0111] , under the constraint of , a target function based on geometric error is constructed:

[0112] (16)

[0113] wherein, is the target function based on geometric error, represents the point mapping under the action of homomorphism transformation. An efficient second-order minimization (ESM) algorithm is adopted to avoid Hessian matrix calculation, improve convergence speed, and solve parameters in the Lie algebra space by iteration:

[0114] (17)

[0115] wherein, is the generator of , and is an optimization variable, representing the linear combination coefficient in the Lie algebra , and the Lie algebra​​ The degree of freedom is 8. The optimized global homography matrix is output , and the preliminary alignment is realized.

[0116] Step 3.3, the target image corresponding to the global homography matrix is divided into several grid sub-regions, and the local homography matrix of each grid sub-region is estimated.

[0117] The global homography transformation The target image corresponding to the global homography transformation is divided into grid sub-regions, and the local homography matrix of each grid sub-region is estimated independently. A joint optimization objective function is constructed, including local alignment error and smoothness constraint:

[0118] (18)

[0119] Wherein, is the joint optimization objective function, is the smoothness constraint for controlling the local transformation, indicates the number of grid sub-regions, and the value range is , is the local homography matrix of the th grid sub-region, indicates the number of pixel points in the th grid sub-region, and the value range is , indicates the total number of feature points in the th grid sub-region, is the adjacent grid set corresponding to the th edge, indicates the coordinates of the th original pixel point in the th grid sub-region, indicates the coordinates of the pixel point matched with the th original pixel point in the target image in the th grid sub-region, is the local homography matrix , is the transformation result of is the total number of adjacent sub-region pairs (edges), is the th grid sub-region in the adjacent grid set , is the local homography matrix of the th grid sub-region, indicates the cross-sub-region matching feature points corresponding to the th edge, which is used to calculate the smoothness constraint. In ESM algorithm is used to iteratively optimize on the manifold, ensuring local alignment accuracy and global smoothness.

[0120] Step 3.4, extract global similarity transformation from global homography matrix, and interpolate local homography matrix and global similarity transformation on the manifold of the general linear group to obtain the interpolation transformation matrix of each grid sub-region.

[0121] Global similarity transformation is a shape-preserving geometric transformation that only contains rotation, translation and uniform scaling, used to constrain deformation in non-overlapping regions. To alleviate the distortion caused by local transformation, the embodiment interpolates the homography matrix of each grid sub-region and the global similarity transformation on the manifold of the general linear group :

[0122] (19)

[0123] where the weight is adaptively adjusted according to the projection position of the grid center in the image, and the dynamic optimization of the parameters is realized through the conventional closed-loop feedback according to the image stitching accuracy, ensuring that the overlapping region retains high alignment accuracy and the non-overlapping region maintains shape consistency, is the interpolation transformation matrix of the first grid sub-region.

[0124] Step 3.5, realize the fusion of adjacent view images by fusing multiple interpolation transformation matrices between adjacent view images, fuse each adjacent two view images, finally output a fused image, and complete the fusion of multiple view images.

[0125] According to the interpolation transformation matrix after interpolation fusion, the vertices in the image grid sub-region are remapped, and the final unified transformation field is obtained by using the weighted average fusion strategy between all grid sub-regions, and then the high-precision image stitching is completed. The overlapping region uses multi-band fusion to eliminate the seam, and the non-overlapping region maintains the original transformation to avoid distortion. The seamless stitched image is generated, which maintains the local alignment accuracy while ensuring the global visual naturalness without distortion. After fusion, the image maintains seamless transition in the overlapping region, minimizes structural distortion in the non-overlapping region, and improves the global visual consistency.

[0126] Step 4, fusion optimization is performed on the whole fused image and the overlapping region of the adjacent two-view images, to obtain a preliminary spliced image. After completing the manifold optimization splicing based on the matrix Lie group, in order to further improve the visual naturalness and transition smoothness of the image fusion region, the application designs a fusion optimization and imaging consistency enhancement mechanism, fully considering the spatial structure relationship of the overlapping region, multi-scale fusion processing, local contrast consistency and cross-view color mapping relationship.

[0127] Step 4.1, align the adjacent view images to the same coordinate system according to the difference transformation matrix between the adjacent view images, and calculate the dynamic fusion weight between each pair of matching pixel points in the overlapping region of the adjacent view images, and update the pixel value of the pixel point in the overlapping region of the fused image based on the dynamic fusion weight.

[0128] Based on the final interpolation transformation matrix obtained in step 3 , after aligning the adjacent view images to the same reference coordinate system, for each pixel point in the overlapping region, a dynamic fusion weight map is constructed to guide the bilateral image fusion. The weight is determined by the normalized Euclidean distance of the pixel point to the non-overlapping boundary of the adjacent image, and the dynamic fusion weight The specific expression of the calculation is as follows:

[0129] (20)

[0130] Wherein, and respectively represent the minimum Euclidean distance of the pixel point to the non-overlapping boundary of the left and right images. This mechanism is complementary to the consistency constraint of the local homography transformation on the adjacent grid boundary in step 3.3 in space, and the final fusion result of the overlapping region is:

[0131] (21)

[0132] Wherein, is the fused pixel value of the pixel point , represents the pixel value of the pixel point on the left image, represents the pixel value of the pixel point on the right image.

[0133] Step 4.2, the updated fusion image is corrected by using a multi-scale Gaussian pyramid.

[0134] In order to further eliminate the seam edge marks caused by local brightness difference or parallax, a three-layer Gaussian pyramid multi-scale fusion strategy is introduced on the basis of step 4.1. The pyramid decomposition layers are constructed for the left and right images:

[0135] (22)

[0136] wherein, is the original input image (one of the left or right image, such as the left or right image), is the initial layer of the pyramid decomposition, is the 0th layer of the three-layer Gaussian pyramid, is the 0th layer of the three-layer Laplacian pyramid, is the 0th layer of the three-layer Gaussian pyramid, is the 0th layer of the three-layer Laplacian pyramid, is the 0th layer of the three-layer Gaussian pyramid, is the Gaussian blur operation.

[0137] The weight-weighted fusion and boundary smoothing operations are independently performed on each layer image, and the final fused image is obtained by layer-by-layer pyramid reconstruction. This method is consistent with the multi-scale geometric feature construction in step 2.1 in the scale space, effectively suppressing the seam diffusion phenomenon caused by multi-scale estimation.

[0138] Step 4.3, perform adaptive histogram equalization processing and color mapping correction based on matching regions on the corrected fused image to obtain a preliminary spliced image.

[0139] (1) Imaging consistency enhancement based on local adaptive histogram equalization. Due to the difference in light sensitivity characteristics of multiple cameras in step 1, the overall brightness and local contrast of the image are inconsistent, which is easy to cause gray level jump in the fusion area. Therefore, this step applies a limited contrast adaptive histogram equalization (CLAHE) processing on the fused image. By default, take 32x32 pixels as a unit block, calculate the local histogram and add a limited contrast:

[0140] (23)

[0141] wherein, is the probability of pixels with a gray value of in the original histogram, is the probability of pixels with a gray value of after adaptive histogram equalization processing, is the probability of pixels with a gray value of in the original histogram, is the control parameter, is the total number of pixels. This method suppresses local overexposure while preserving details, forming a closed-loop compensation mechanism with the initial preprocessing in step 1, significantly improving the gray uniformity of the spliced image.

[0142] (2) Construction and application of color mapping function based on matching region. To solve the color gamut difference problem of multi-view camera imaging in step 1, the color mapping function of the matching point pair is constructed in the Lab color space after adaptive histogram equalization processing based on the region feature matching result in step 3. By statistically calculating the mean and standard deviation of the matching points in the source image and the target image, the following linear mapping relationship is defined:

[0143] (24)

[0144] Wherein, the subscripts respectively represent the source image and the target image, represents the brightness value of the source image, represents the color channel of the source image, representing the red-green opposite color, represents the color channel of the source image, representing the yellow-blue opposite color, represents the brightness value of the target image, represents the color channel of the target image, representing the red-green opposite color, represents the color channel of the target image, representing the yellow-blue opposite color, represents the mean value of the source image in the Lab three channels, represents the mean value of the target image in the Lab three channels, represents the standard deviation of the source image in the Lab three channels, represents the standard deviation of the target image in the Lab three channels. The mapping significantly alleviates the fusion jump caused by the inconsistent color response between multiple cameras, and enhances the overall tone consistency of the stitched image.

[0145] Step 5, quality evaluation of the preliminary stitched image, if the quality evaluation fails, re-perform the fusion; otherwise, output the preliminary stitched image as the final surround view panoramic image obtained by surround view stitching. To ensure that the image stitching method has stable and reliable stitching quality in different scenes, the embodiment further proposes an image quality feedback mechanism based on multi-index evaluation and adaptive control, realizing real-time closed-loop adjustment optimization of the stitching system.

[0146] Step 5.1, calculation of stitching image quality evaluation index. After each frame of image stitching is completed, the following three core indexes are calculated in real time.

[0147] (1) Structural Similarity Index (SSIM): used to measure the structural fidelity of the overlapping area fusion effect, and the calculation formula is as follows:

[0148] (25)

[0149] Wherein, the parameter , parameters , for image pixel dynamic range, represent mean and variance, respectively, mean of image , respectively, standard deviation of image , represent covariance of image , represent two images to be compared, in this embodiment, the original sub-image in the overlap area and the fused sub-image. This index is directly related to the fusion effect of the Gaussian pyramid in step 4.2.

[0150] (2) Root Mean Square Error (RMSE): used to evaluate the registration accuracy based on the global homography matrix in step 3.2:

[0151] (26)

[0152] wherein, and respectively represent the pixel coordinates before and after matching, reflecting the constraint ability of the manifold optimization on the geometric alignment accuracy in step 3.2.

[0153] (3) Correct Matching Rate (CMR): used to quantify the accuracy of regional feature matching in step 3.1, and the calculation method is:

[0154] (27)

[0155] wherein, represents the number of matches consistent with manual verification annotation, determined by high-precision prior data, is the total number of matching points.

[0156] Step 5.2, dynamic parameter self-adaptive adjustment based on index mapping. According to the above quality indexes, the parameter-performance mapping relationship is constructed, and the parameters of the key modules are automatically adjusted for the next moment of the surround stitching task to realize the adaptive optimization of the system: when (the first threshold value), the contrast suppression coefficient in the CLAHE processing in step 4.3 is enhanced to improve the local detail performance; when pixels (third threshold value), the inlier judgment threshold of the RANSAC method in step 3.1 is tightened to improve the robustness of geometric constraints; when (the fourth threshold value), the confidence threshold of feature matching in step 3.1 is increased to strengthen the matching accuracy.

[0157] ​Step 5.3, the contrast enhancement intensity in the adaptive histogram equalization process is adjusted by a PID controller, the error signal of the PID controller is defined as the difference between the preset structure similarity index of the preliminary splicing image and the actual structure similarity index of the preliminary splicing image, dynamic closed-loop adjustment of quality control is realized, and the control law is expressed as:

[0158] (28)

[0159] wherein, is a proportional coefficient, is an integral coefficient, is a differential coefficient, and the error signal is defined as , represents the preset image splicing quality ideal value of the system, represents the current frame image splicing quality evaluation result calculated in real time, and the PID output is used for adjusting the contrast enhancement intensity in the CLAHE method. This mechanism cooperates with the color correction function in step 4.3 to form a synergistic optimization, and improves the perceptual consistency and visual quality of the fusion region.

[0160] Step 5.4, abnormal image processing and robust control strategy. For sudden occlusion, sudden change of light or camera deviation caused splicing anomaly, the following abnormal response mechanism is designed in this embodiment: when the number of pixels in the continuous 3 frame image splicing results is greater than the fifth threshold value (the third threshold value is less than the fifth threshold value), it means that the quality evaluation fails, and it is returned to step 3 to execute again; when the value instantaneously drops to 0.15 or less (that is, the structure similarity index of the preliminary splicing image is less than the second threshold value (such as 0.15), the first threshold value is greater than the second threshold value, and the difference between the current structure similarity index and the last structure similarity index is greater than the drop threshold value (such as 0.5)), it means that the quality evaluation fails, and it is returned to step 4.2 to execute again; in addition, this embodiment can monitor the external camera data anomaly in real time by means of the residual statistics of the RANSAC method in the feature matching stage of step 3.1, such as frame synchronization signal loss, camera data interruption, etc., to automatically trigger system alarm. After the current time is executed, it is judged whether to continue to execute the ring-view splicing task, if yes, it enters the next time to continue to collect data for processing; otherwise, the task is exited and ended.

[0161] The surround view stitching method provided by the embodiment can perform high-precision registration and seamless fusion on images collected by multiple wide-angle cameras, significantly improves the stitching quality and processing efficiency of panoramic images, and provides more accurate and stable environment perception support for intelligent driving systems. The method introduces a feature extraction mechanism based on multivariate Laplace distribution and Lie group embedding, combines a transformation optimization strategy under manifold constraint and a pyramid fusion method, realizes the target of robust stitching performance in complex lighting, occlusion and disparity scenes, and effectively breaks through the technical bottlenecks of traditional schemes in terms of accuracy, real-time performance and consistency.

[0162] Embodiment 2

[0163] The embodiment provides a surround view stitching device based on geometric Lie group feature extraction and manifold optimization, which comprises a processor and a memory storing a plurality of computer instructions, and the computer instructions are executed by the processor to realize the steps of the surround view stitching method based on geometric Lie group feature extraction and manifold optimization.

[0164] The specific limitations of the surround view stitching device based on geometric Lie group feature extraction and manifold optimization can be referred to the limitations of the surround view stitching method based on geometric Lie group feature extraction and manifold optimization in the above, and will not be repeated here.

[0165] The memory and the processor are directly or indirectly electrically connected to realize the transmission or interaction of data. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines. The memory stores a computer program that can run on the processor, and the processor realizes the method by running the computer program stored in the memory.

[0166] The memory can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), etc. The memory is used to store programs, and the processor executes the programs after receiving an execution instruction.

[0167] The processor can be an integrated circuit chip with data processing capability. The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc. The processor can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0168] Embodiment 3

[0169] The embodiment provides a surround view stitching device based on geometric Lie group feature extraction and manifold optimization. Based on a geometric Lie group feature extraction and optimization strategy, image data collected by multiple vehicle-mounted cameras are processed and analyzed in real time, including the following functional modules.

[0170] An image acquisition module is configured to synchronously acquire original image data collected by six wide-angle cameras, including high dynamic range RGB images, lens distortion parameters, frame timestamp synchronization signals, and internal and external parameter calibration matrix data of the cameras. The module relies on a hardware synchronization triggering mechanism to ensure the spatiotemporal consistency of image acquisition.

[0171] A feature processing module is configured to extract and construct regional geometric feature descriptors with Lie group structure. The module extracts multi-dimensional features of local regions on a multi-scale image pyramid, constructs three types of descriptors based on multivariate Laplace modeling, and completes feature vectorization based on Lie group embedding and logarithmic mapping. Through matching confidence and geometric consistency double constraints, cross-view regional level robust matching is realized.

[0172] An image processing module is configured to complete accurate registration and seamless fusion between images. The module estimates global and local transformation matrices (R, T) on the Lie group manifold, realizes deformation distortion control and multi-view overlap alignment by polar decomposition interpolation fusion of grid region embedding matrices, and introduces a Laplacian pyramid fusion strategy and a color mapping correction algorithm to ensure the structural consistency and color consistency of the fused image.

[0173] A quality evaluation module is configured to monitor the quality indicators of the stitching results in real time, including structural similarity index (SSIM), root mean square error (RMSE), and correct matching rate (CMR). The module feeds back the key indicators to the system adjustment module to form a closed-loop quality control process.

[0174] ​A dynamic parameter self-adaptive adjustment module is configured to self-adaptively adjust the processing flow parameters according to the quality evaluation result, including dynamically adjusting the contrast enhancement factor of CLAHE, the inlier threshold of RANSAC homography transformation, the matching feature ratio threshold, and the like, so as to guarantee the robustness and stability of the system under different road scenes and light conditions.

[0175] The specific limitations of the surround view stitching device regarding geometric Lie group feature extraction and manifold optimization can be found in the limitations of the surround view stitching method of geometric Lie group feature extraction and manifold optimization described above, which will not be repeated here.

[0176] Embodiment 4:

[0177] As shown in Figure 3 The embodiment provides a complete driving system, which comprises a multi-channel image acquisition device, an image data processing and analysis device, a data information recording device, a vehicle control unit, a data communication device, and a terminal device.

[0178] The multi-channel image acquisition device is configured to acquire original images with overlapping viewing angles, and comprises six 2 million-pixel fisheye cameras arranged in a ring shape. The multi-channel video is synchronously acquired through a hardware synchronous triggering mechanism, HDR imaging and automatic exposure control are supported, and the acquired original video data is attached with a timestamp and calibration parameters.

[0179] The image data processing and analysis device is configured to execute the steps of the surround view stitching method of geometric Lie group feature extraction and manifold optimization. The specific limitations of the surround view stitching method of geometric Lie group feature extraction and manifold optimization are described in Embodiment 1, which will not be repeated here.

[0180] The data information recording device is configured to store the original video data and the processed panoramic image data acquired by the system in real time, record the system running state log and key event information, and provide data support for subsequent data analysis and system optimization.

[0181] The vehicle control unit is configured to receive the panoramic visual information generated by the image data processing and analysis device, and interact with other control systems of the vehicle to provide environmental perception input for the automatic driving or advanced driving assistance system.

[0182] The data communication device is configured to establish a high-speed data transmission channel between the devices to ensure real-time and reliable transmission of video data, control signals and state information, and provide support for multiple communication protocols to meet the needs of different application scenarios.

[0183] The terminal device is configured to display the processed panoramic surround view image to the driver in real time, and provide a human-computer interaction interface to support user operation control and parameter setting of the system, and receive and display system warning and prompt information.

[0184] Any combination of the technical features in the above-described embodiments can be made, and for the sake of brevity, not all possible combinations are described, however, as long as there is no conflict, any combination of the technical features should be considered within the scope of the present disclosure.

[0185] The above-described embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the scope of the application. It should be pointed out that, for ordinary skilled persons in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for panoramic stitching based on geometric Lie group feature extraction and manifold optimization, characterized in that, The geometric Lie group feature extraction and manifold optimization-based surround view stitching method comprises: A multi-path image acquisition device in a ring array is used to acquire multi-path perspective images; For each perspective image, a regional geometric feature description method based on Lie group structure modeling is used to construct a geometric feature descriptor; Based on the geometric feature descriptors corresponding to the perspective images, a manifold stitching optimization method based on matrix Lie group structure is used to complete multi-path perspective image fusion; The overall fused image and the overlapping region of adjacent two perspective images are fused and optimized to obtain a preliminary stitched image; The preliminary stitched image is quality evaluated, if the quality evaluation fails, the fusion is re-performed, otherwise, the preliminary stitched image is output as a surround view panorama image obtained by final surround view stitching; The regional geometric feature description method based on Lie group structure modeling comprises: The five-layer Gaussian pyramid structure is adopted to blur and down-sample the perspective image, and each layer of the Gaussian pyramid generates a Gaussian blurred image, is a number of sub-layers of the five-layer Gaussian pyramid structure; A plurality of Gaussian blur images are selected from each Gaussian pyramid, and feature matrices of the selected Gaussian blur images are extracted; Type II multi-Laplace distribution parameters of each feature matrix of each Gaussian blur image are estimated, and the estimated Type II multi-Laplace distribution parameters are embedded into a Lie group structure to obtain an affine embedding matrix, and a first type of feature descriptor is obtained by logarithmic mapping of the affine embedding matrix; A Lie group quotient structure optimization strategy is introduced to perform polar decomposition on the affine embedding matrix to obtain a left polar positive definite matrix and a right polar positive definite matrix, and a second type of feature descriptor and a third type of feature descriptor are obtained by logarithmic mapping of the left polar positive definite matrix and the right polar positive definite matrix, respectively; The manifold stitching optimization method based on matrix Lie group structure comprises: Based on the geometric feature descriptors of adjacent perspective images, feature matching is performed on the adjacent perspective images, and false matches are removed by RANSAC to obtain a matching pixel point set; Based on the matching pixel set, Lie groups are performed on images from adjacent viewpoints. Global alignment optimization is performed on the surface to obtain the global homography matrix; A target image corresponding to a global homography matrix is divided into a plurality of grid sub-regions, and a local homography matrix of each grid sub-region is estimated; extract global similarity transformation from global homography matrix and interpolate local homography matrix and global similarity transformation in the generalized linear group manifold to obtain the interpolation transformation matrix of each mesh sub-region; A plurality of interpolation transformation matrices between adjacent perspective images are fused to realize fusion of the adjacent perspective images; Each adjacent two perspective images are fused to finally output a fused image, and multi-path perspective image fusion is completed.

2. The geometric Lie group feature extraction and manifold optimization based surround stitching method of claim 1, wherein, The multi-path image acquisition device in a ring array is used to acquire multi-path perspective images, which comprises: A multi-path image acquisition device is used to form a ring array, adjacent two image acquisition devices have overlapping perspectives, and each image acquisition device is calibrated; A unified clock signal source is used to synchronously trigger the multi-path image acquisition device to acquire original images; According to a rigid transformation relationship from a camera coordinate system to a vehicle coordinate system, the original images are converted to the vehicle coordinate system, and the converted images are adaptively white balanced to obtain corrected images; Nonlinear gamma correction processing is introduced to perform nonlinear mapping transformation on the brightness of the corrected images to obtain perspective images.

3. The geometric Lie group feature extraction and manifold optimization based surround stitching method of claim 1, wherein, The overall fused image and the overlapping region of adjacent two perspective images are fused and optimized to obtain a preliminary stitched image, which comprises: According to a difference transformation matrix between adjacent view images, the adjacent view images are aligned to a same coordinate system, and a dynamic fusion weight between each pair of matching pixel points in an overlapping region of the adjacent view images is calculated, and a pixel value of a pixel point in the overlapping region in a fused image is updated based on the dynamic fusion weight; The updated fused image is corrected by using a multi-scale Gaussian pyramid; An adaptive histogram equalization processing and a color mapping correction based on a matching region are performed on the corrected fused image to obtain a preliminary spliced image.

4. The geometric Lie group feature extraction and manifold optimization based surround stitching method of claim 3, wherein, A contrast enhancement intensity in the adaptive histogram equalization processing is adjusted by using a PID controller, and an error signal of the PID controller is defined as a difference between a preset structural similarity index of the preliminary spliced image and an actual structural similarity index of the preliminary spliced image.

5. The geometric Lie group feature extraction and manifold optimization based surround stitching method of claim 3, wherein, The preliminary spliced image is quality evaluated, and if the quality evaluation fails, the fusion is re-performed; Otherwise, the preliminary spliced image is output as a final surround view spliced surround view panoramic image, including: If a root mean square error of a plurality of continuous preliminary spliced images is greater than a fifth threshold value, the quality evaluation fails, and the step of obtaining the matching pixel point set is re-performed; If a structural similarity index of the preliminary spliced image is less than a second threshold value and a difference between the current structural similarity index and a last structural similarity index is greater than a falling threshold value, the quality evaluation fails, and the step of correcting the updated fused image by using the multi-scale Gaussian pyramid is re-performed; Otherwise, the quality evaluation passes, and the preliminary spliced image is output as the final surround view spliced surround view panoramic image.

6. The geometric Lie group feature extraction and manifold optimization based surround stitching method of claim 3, wherein, The surround view splicing method based on the geometric Lie group feature extraction and the manifold optimization further includes dynamic parameter adaptive adjustment, specifically: If the structural similarity index of the preliminary spliced image is less than a first threshold value, a contrast suppression coefficient in the adaptive histogram equalization processing is enhanced; If the root mean square error of the preliminary spliced image is greater than a third threshold value, an inlier judgment threshold value of the RANSAC method is reduced; If a correct matching rate of the preliminary spliced image is less than a fourth threshold value, a confidence threshold value of the RANSAC method is increased.

7. A driving system characterized by comprising: The driving system includes a multi-path image acquisition device, an image data processing and analysis device, a data information recording device, a vehicle control unit, a data communication device, and a terminal device, wherein: The multi-path image acquisition device is configured to acquire original images with overlapping view angles; The image data processing and analysis device is configured to perform steps of the surround view splicing method based on the geometric Lie group feature extraction and the manifold optimization in any one of claims 1 to 6; The data communication device is configured to realize data transmission between the multi-path image acquisition device and the image data processing and analysis device; The data information recording device is configured to record data related to the surround view splicing method based on the geometric Lie group feature extraction and the manifold optimization; The vehicle control unit is configured to control a vehicle in combination with the final surround view panoramic image; The terminal device is configured to display the surround view panoramic image.

Images