A Method and System for Generating 360° Images of Subway Vehicles Based on Image Stitching

Abstract

This invention belongs to the field of image generation, specifically relating to a method and system for generating 360° images of subway vehicles based on image stitching. The method includes: acquiring source images collected along the subway vehicle; obtaining effective matching point pairs through feature extraction matching and spatial topological similarity filtering; selecting the camera pair with the strongest geometric constraints as a benchmark; using incremental triangulation to recover the camera pose and 3D point cloud; in global bundle adjustment, optimization is performed by combining reprojection error and linear regularization terms based on the geometric principal axes of principal component analysis, with the regularization term weights adjusted according to the average distance from the point cloud to the principal axes; after optimization, the source images are projected onto a cylindrical canvas, and the optimal image gradient is selected based on gradient information entropy to construct a gradient field; a seamless 360° panoramic image is generated by solving the Poisson equation. This invention can effectively eliminate mismatches, suppress 3D reconstruction offset, and generate high-quality seamless panoramic images.

Description

Technical Field

[0001] This invention belongs to the field of image generation, specifically relating to a method and system for generating 360° images of subway vehicles based on image stitching. Background Technology

[0002] In scenarios such as rail transit operation and maintenance, and vehicle exterior quality inspection, 360° panoramic images of subway vehicles can completely present the surface condition of the vehicle body, providing intuitive and comprehensive visual evidence for defect detection, paint evaluation, and fault tracing. Therefore, the generation technology has significant engineering application value. Currently, the mainstream technical approach for generating 360° panoramic images of large objects is based on a combination of Structure from Motion (SfM) and image stitching. This method typically uses a mobile device to acquire a series of overlapping source images along the object, and then uses algorithms such as Scale Invariant Feature Transform (SIFT) to extract and match feature points between the images to establish geometric correspondences between them. Subsequently, using the SfM process, the poses of all cameras and the sparse 3D point cloud structure of the scene are recovered from the matched feature points using incremental or global methods. Finally, all source images are projected onto a preset canvas according to the calculated camera poses, and the overlapping areas are processed by image fusion algorithms to generate a complete panoramic image.

[0003] However, subway vehicles possess two significant characteristics: a long, narrow linear structure and highly repetitive textures. The vehicle body exhibits a regular linear distribution with numerous repetitive elements on its surface, such as continuously arranged windows, uniformly sized doors, and periodic paint stripes. The core logic of traditional feature matching algorithms is to determine matching relationships based on the similarity of feature descriptors. However, the repetitive texture of subway vehicle bodies leads to highly similar descriptors generated by feature points at different locations, resulting in a large number of mismatches. Conventional mismatch removal strategies, such as the Random Sample Consensus (RANSAC) algorithm, rely on the premise that the proportion of correct matches is higher than that of mismatches. When the proportion of mismatches is too high, its performance degrades sharply. If subsequent 3D reconstruction is based on mismatches, it will directly cause camera pose calculation errors, creating a potential accuracy problem for panoramic generation. Furthermore, to cover the entire length of the subway vehicle, the shooting path needs to move linearly and uniformly along the vehicle body. This movement method is a degenerate configuration in SfM technology, meaning the camera only moves in a single direction, resulting in insufficient projection constraints of feature points on the image plane and a tendency to accumulate errors. Traditional global bundle adjustment optimization (BBEM) focuses solely on minimizing 3D point reprojection errors, lacking constraints on the global geometry of the scene, such as the linear physical shape of subway vehicles. During iterative optimization, accumulated errors amplify, leading to curved or offset camera trajectories in the reconstructed camera paths and stretched or deformed 3D point clouds. This ultimately causes spatial misalignment when the source image is projected onto the canvas, affecting the structural consistency of the panoramic image. Furthermore, during image fusion, processing misalignments caused by pose errors or parallax often results in visual imperfections such as blurring and ghosting at the stitching seams, impacting the overall quality and detail of the panoramic image. Summary of the Invention

[0004] Therefore, the purpose of this invention is to propose a method and system for generating 360° images of subway vehicles based on image stitching, in order to solve the technical problems of high feature point mismatch rate, easy drift and deformation in motion recovery structure reconstruction, and easy visual defects in the final image fusion when the existing technology processes large objects such as subway cars with highly repetitive textures.

[0005] To address the above problems, the technical solution of the method for generating 360° images of subway vehicles based on image stitching proposed in this invention is as follows:

[0006] A method for generating 360° images of subway vehicles based on image stitching includes the following steps:

[0007] Acquire source images along the subway vehicle, extract and match feature points between the source images to form an initial set of matching point pairs; calculate the spatial topological similarity of the two feature points of the matching point pairs in the initial set of matching point pairs in their respective local neighborhoods as the cohesion score, and filter the matching point pairs with scores greater than a preset threshold to form an effective set of matching point pairs.

[0008] Based on the set of effective matching point pairs, the camera pair with the strongest geometric constraints is selected as the initial camera pair by evaluating the baseline length, number and distribution of each camera pair. Using the initial camera pair as a reference, the pose of the remaining cameras and the initial 3D point cloud are incrementally recovered using the triangulation method.

[0009] The objective function is set as a weighted sum of the reprojection error term and the linear regularization term. The geometric principal axes of the current 3D point cloud are determined, and the weighted sum of the Euclidean distances from all 3D points in the 3D point cloud to the geometric principal axes is calculated as the linear regularization term. The weights of the linear regularization term are adjusted according to the average distance from all 3D points to the geometric principal axes, and the geometric principal axes are periodically updated during the optimization process. Global bundle adjustment optimization is performed to minimize the objective function.

[0010] All source images are projected onto a cylindrical canvas according to the optimized camera pose. Based on the gradient information entropy of each pixel on the cylindrical canvas in the overlapping area of ​​the source image projection, the gradient information of the optimal source image is selected to construct a global gradient field. A seamless 360° panoramic image is obtained by solving the Poisson equation.

[0011] Furthermore, the method for constructing the set of valid matching point pairs is as follows:

[0012] For each matching point pair, a set number of feature points within the image containing the feature point are selected as a local neighborhood point set.

[0013] A spatial topology graph is constructed with the feature point as the center and the normalized distance and angle between the feature point and the set number of feature points as the sides.

[0014] By calculating the graph editing distance between the two spatial topological graphs of the matching point pair, a structural similarity score ranging from 0 to 1 is obtained, and the structural similarity score is used as the cohesion score.

[0015] Set a preset threshold for cohesion score, and retain matching point pairs with scores greater than the preset threshold to form a set of valid matching point pairs.

[0016] Furthermore, the method for selecting the initial camera pair is as follows:

[0017] For any camera pair, the evaluation is performed by calculating the geometric constraint strength score of the camera pair, whereby the geometric constraint strength score is defined as... The calculation formula is as follows:

[0018] ;

[0019] In the formula, This represents the number of valid matching point pairs between two cameras in a camera pair. The disparity between the effective matching point pairs between the two cameras in a camera pair is calculated as the median pixel distance between the corresponding feature points in their respective image coordinate systems. The area of ​​the convex hull formed by the valid matching points on one of the images; , , These are the maximum values ​​of the corresponding indicators for all camera pairs; , , These are the preset weights for the corresponding indicators, and they satisfy... ;

[0020] Calculate the geometric constraint strength score of all camera pairs, and select the camera pair with the largest geometric constraint strength score as the initial camera pair.

[0021] Furthermore, the incremental recovery of the pose of the remaining cameras using the triangulation method includes: selecting a camera from the unprocessed cameras that shares the most effective matching point pairs with the cameras whose poses have been recovered as the next camera to be processed; using the correspondence between the two-dimensional feature points in the camera and the existing three-dimensional point cloud, solving the pose of the camera using the perspective n-point method combined with the RANSAC algorithm.

[0022] Furthermore, the geometric principal axes are determined using principal component analysis, including:

[0023] Construct a coordinate matrix from the coordinates of all 3D points in the current 3D point cloud;

[0024] Calculate the covariance matrix of the coordinate matrix;

[0025] The covariance matrix is ​​decomposed into eigenvalues ​​to obtain three eigenvalues ​​and three corresponding eigenvectors.

[0026] The eigenvector corresponding to the largest eigenvalue is selected as the direction vector of the geometric principal axis.

[0027] Furthermore, the method for adjusting the weights of the linear regularization term is as follows:

[0028] Before starting global bundle adjustment optimization, calculate the average Euclidean distance from all 3D points to the initial geometric principal axes, and use this average distance as the initial average distance. ;

[0029] Before each iteration of optimization, the average Euclidean distance from all 3D points to the current geometric principal axis is calculated and used as the current average distance. ;

[0030] The adjustment formula for the weight of the linear regularization term is: ;In the formula, The preset initial weighting coefficients, The weights are the linear regularization terms.

[0031] Furthermore, the method for constructing the global gradient field is as follows:

[0032] For any pixel on the cylindrical canvas, find all the source images that can be projected onto that pixel; for each source image in the source image set, extract a neighborhood window of a preset size at the projection position of the pixel.

[0033] Calculate the gradient magnitude of the image within the neighborhood window, represent the gradient magnitude as 256 levels, and plot a histogram; calculate the gradient information entropy of the neighborhood window based on the histogram, and select the source image with the largest gradient information entropy as the optimal source image of the pixel.

[0034] Extract the horizontal and vertical gradient components of the optimal source image at the specified pixel to construct a global gradient field.

[0035] Furthermore, the method for reconstructing a seamless 360° panoramic image by solving the Poisson equation is as follows:

[0036] The divergence is calculated based on the constructed global gradient field;

[0037] Solve the Poisson equation by treating the divergence field as a right-hand side term. Where I is the intensity of the panoramic image to be solved, and G is the global gradient field. For the Laplace operator, For divergence operators;

[0038] Set Dirichlet boundary conditions to force the pixel intensity values ​​at the left and right edges of the cylindrical canvas to be equal;

[0039] The Poisson equation is solved using a solver based on Fast Fourier Transform to obtain a 360° panoramic image.

[0040] The technical solution of the 360° image generation system for subway vehicles based on image stitching proposed in this invention is as follows:

[0041] A 360° image generation system for subway vehicles based on image stitching, characterized in that it includes:

[0042] The filtering module is used to acquire source images collected along the subway vehicle, extract and match feature points between the source images to form an initial set of matching point pairs; calculate the spatial topological similarity of the two feature points of the matching point pairs in the initial set of matching point pairs in their respective local neighborhoods as the cohesion score, and filter the matching point pairs with scores greater than a preset threshold to form an effective set of matching point pairs.

[0043] The recovery module is used to select the camera pair with the strongest geometric constraints as the initial camera pair based on the set of effective matching point pairs by evaluating the baseline length, number and distribution of each camera pair; and incrementally recover the pose of the remaining cameras and the initial 3D point cloud using triangulation methods based on the initial camera pair.

[0044] The update module is used to set the objective function as a weighted sum of the reprojection error term and the linear regularization term, determine the geometric principal axes of the current 3D point cloud, and calculate the weighted sum of the Euclidean distances from all 3D points in the 3D point cloud to the geometric principal axes as the linear regularization term; adjust the weight of the linear regularization term according to the average distance from all 3D points to the geometric principal axes, and periodically update the geometric principal axes during the optimization process; perform global bundle adjustment optimization to minimize the objective function.

[0045] The reconstruction module projects all source images onto a cylindrical canvas according to the optimized camera pose. Based on the gradient entropy of each pixel on the cylindrical canvas in the overlapping area of ​​the source image projection, it selects the gradient information of the optimal source image to construct a global gradient field. By solving the Poisson equation, a seamless 360° panoramic image is reconstructed.

[0046] Furthermore, the method for constructing the set of valid matching point pairs is as follows:

[0047] For each matching point pair, a set number of feature points within the image containing the feature point are selected as a local neighborhood point set.

[0048] A spatial topology graph is constructed with the feature point as the center and the normalized distance and angle between the feature point and the set number of feature points as the sides.

[0049] By calculating the graph editing distance between the two spatial topological graphs of the matching point pair, a structural similarity score ranging from 0 to 1 is obtained, and the structural similarity score is used as the cohesion score.

[0050] Set a preset threshold for cohesion score, and retain matching point pairs with scores greater than the preset threshold to form a set of valid matching point pairs.

[0051] Furthermore, the method for selecting the initial camera pair is as follows:

[0052] For any camera pair, the evaluation is performed by calculating the geometric constraint strength score of the camera pair, whereby the geometric constraint strength score is defined as... The calculation formula is as follows:

[0053] ;

[0054] In the formula, This represents the number of valid matching point pairs between two cameras in a camera pair. The disparity between the effective matching point pairs between the two cameras in a camera pair is calculated as the median pixel distance between the corresponding feature points in their respective image coordinate systems. The area of ​​the convex hull formed by the valid matching points on one of the images; , , These are the maximum values ​​of the corresponding indicators for all camera pairs; , , These are the preset weights for the corresponding indicators, and they satisfy... ;

[0055] Calculate the geometric constraint strength score of all camera pairs, and select the camera pair with the largest geometric constraint strength score as the initial camera pair.

[0056] Furthermore, the incremental recovery of the pose of the remaining cameras using the triangulation method includes: selecting a camera from the unprocessed cameras that shares the most effective matching point pairs with the cameras whose poses have been recovered as the next camera to be processed; using the correspondence between the two-dimensional feature points in the camera and the existing three-dimensional point cloud, solving the pose of the camera using the perspective n-point method combined with the RANSAC algorithm.

[0057] Furthermore, the geometric principal axes are determined using principal component analysis, including:

[0058] Construct a coordinate matrix from the coordinates of all 3D points in the current 3D point cloud;

[0059] Calculate the covariance matrix of the coordinate matrix;

[0060] The covariance matrix is ​​decomposed into eigenvalues ​​to obtain three eigenvalues ​​and three corresponding eigenvectors.

[0061] The eigenvector corresponding to the largest eigenvalue is selected as the direction vector of the geometric principal axis.

[0062] Furthermore, the method for adjusting the weights of the linear regularization term is as follows:

[0063] Before starting global bundle adjustment optimization, calculate the average Euclidean distance from all 3D points to the initial geometric principal axes, and use this average distance as the initial average distance. ;

[0064] Before each iteration of optimization, the average Euclidean distance from all 3D points to the current geometric principal axis is calculated and used as the current average distance. ;

[0065] The adjustment formula for the weight of the linear regularization term is: ;In the formula, The preset initial weighting coefficients, The weights are the linear regularization terms.

[0066] Furthermore, the method for constructing the global gradient field is as follows:

[0067] For any pixel on the cylindrical canvas, find all the source images that can be projected onto that pixel; for each source image in the source image set, extract a neighborhood window of a preset size at the projection position of the pixel.

[0068] Calculate the gradient magnitude of the image within the neighborhood window, represent the gradient magnitude as 256 levels, and plot a histogram; calculate the gradient information entropy of the neighborhood window based on the histogram, and select the source image with the largest gradient information entropy as the optimal source image of the pixel.

[0069] Extract the horizontal and vertical gradient components of the optimal source image at the specified pixel to construct a global gradient field.

[0070] Furthermore, the method for reconstructing a seamless 360° panoramic image by solving the Poisson equation is as follows:

[0071] The divergence is calculated based on the constructed global gradient field;

[0072] Solve the Poisson equation by treating the divergence field as a right-hand side term. Where I is the intensity of the panoramic image to be solved, and G is the global gradient field. For the Laplace operator, For divergence operators;

[0073] Set Dirichlet boundary conditions to force the pixel intensity values ​​at the left and right edges of the cylindrical canvas to be equal;

[0074] The Poisson equation is solved using a solver based on Fast Fourier Transform to obtain a 360° panoramic image.

[0075] The beneficial effects of this invention are as follows: This invention filters feature matching point pairs through spatial topological graph structural similarity, which can eliminate mismatches caused by repetitive textures on the surface of subway vehicles. In the global bundle adjustment optimization, a linear regularization term based on the geometric principal axis is used. This linear regularization term, as a prior knowledge constraint, can strongly suppress the drift and deformation of the camera trajectory and 3D point cloud caused by the narrow shooting path, ensuring that the reconstruction result macroscopically conforms to the linear physical shape of the subway vehicle. In the image fusion stage, a strategy based on gradient information entropy selection and Poisson equation reconstruction is adopted, which can retain the clearest texture details in the source image and generate seamless, ghost-free, high-quality panoramic images, avoiding the stitching defects common in traditional fusion algorithms. Attached Figure Description

[0076] Figure 1 This is a flowchart illustrating the steps of the method for generating 360° images of subway vehicles based on image stitching according to the present invention.

[0077] Figure 2 This is a schematic diagram illustrating the determination of geometric principal axes using principal component analysis. Detailed Implementation

[0078] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0079] Specific embodiments of the method for generating 360° images of subway vehicles based on image stitching proposed in this invention:

[0080] like Figure 1 As shown, the method for generating 360° images of subway vehicles based on image stitching includes the following steps:

[0081] S1. Acquire source images collected along the subway vehicle, extract and match feature points between source images to form an initial set of matching point pairs; calculate the spatial topological similarity of the two feature points of the matching point pairs in the initial set of matching point pairs in their respective local neighborhoods as the cohesion score, and filter matching point pairs with scores greater than a preset threshold to form an effective set of matching point pairs.

[0082] Specifically, operators hold a camera or smartphone and walk parallel to one side of the subway car at a constant speed, continuously capturing video or photos at a fixed frame rate to ensure that there is more than 60% overlap between adjacent images, forming a source image sequence. For each image in the source image sequence, the Scale Invariant Feature Transform (SIFT) algorithm is used to detect stable keypoints such as corners and blobs as feature points, and a 128-dimensional representation vector is generated for each feature point. A fast nearest neighbor search algorithm based on FLANN is used to match the feature point representations of any two images, and combined with the Lowe ratio test (the ratio of the nearest neighbor distance to the second nearest neighbor distance is less than a preset threshold), fuzzy matches are eliminated, thus constructing an initial set of matching point pairs containing potentially correct matches and mismatches.

[0083] For any matching point pair in the initial set of matching point pairs, a circular region of radius r is defined in each image as a local neighborhood, centered on the two feature points contained in the matching point pair. All other feature points within this local neighborhood are identified, and the central feature point is connected to each feature point in the local neighborhood to form a set of two-dimensional vectors. The magnitude and angle between each two-dimensional vector and the positive horizontal direction are calculated, forming a local spatial topology graph. By comparing the local spatial topology graphs corresponding to two central feature points—for example, comparing the similarity of the vector magnitude sequences sorted by angle in the two graphs, or calculating the Bach distance of the histogram of the angular distribution of feature points within the local neighborhood—a structural similarity score, i.e., a cohesion score, between 0 and 1 is obtained. A high threshold is set; if the score is greater than this threshold, the matching point pair is considered to be consistent in local geometry and is a high-confidence match, and is retained; otherwise, it is discarded. This process is repeated for all initial matching point pairs to obtain the set of valid matching point pairs.

[0084] In an optional embodiment, the effective set of matching point pairs is constructed as follows:

[0085] For each feature point in a matching point pair, the 10 feature points closest to the feature point in the image where the feature point is located are selected as a local neighborhood point set.

[0086] A spatial topology graph is constructed with the feature point as the center and the normalized distance and angle between the feature point and the above 10 feature points as the edges.

[0087] By calculating the graph editing distance between the two spatial topological graphs of the matching point pair, a structural similarity score ranging from 0 to 1 is obtained, and the structural similarity score is used as the cohesion score.

[0088] Set a preset threshold for cohesion score, and retain matching point pairs with scores greater than the preset threshold to form a set of valid matching point pairs.

[0089] Of course, the number of feature points in the selected local neighborhood point set is not limited to 10. It can be selected according to actual needs, as long as it is sufficient to characterize the stability of the local geometric structure.

[0090] For example, for an initial pair of matching points existing in images A and B, the 10 closest other feature points to one of the feature points are found in image A, forming a local neighborhood. Centered on the feature point, the distances and relative angles from this feature point to the aforementioned 10 feature points are calculated, and the geometric relationships are constructed into a graph structure representing the local spatial topology of the feature point in image A. More specifically, with the feature point as the center node, the 10 closest other feature points in image A are selected as neighboring nodes. The vector from the center node to each neighboring node is calculated, and the length of the vector and its angle with the positive horizontal direction of the image are used as dual attributes of the edge connecting the center and the neighboring node. In one embodiment, the edge includes normalized distance and angle attributes. The constructed graph not only encodes the relative distance and orientation of the neighboring points but also achieves scale invariance through distance normalization, thereby more accurately representing the local spatial structure around the feature point. The same operation is performed on another feature point in image B to obtain a second spatial topology graph.

[0091] By calculating the graph edit distance, we can assess the minimum operational cost required to transform one spatial topology into another, and convert this minimum operational cost into a similarity score between 0 and 1. If the structures of two local regions are almost identical, their graph structures are very similar. Conversely, if a match is incorrect, the neighborhood structures around the two feature points will be very different. A threshold is set, and only matching point pairs with scores higher than this threshold are retained; these are considered highly cohesive and reliable matches, thus forming a set of effective matching point pairs.

[0092] S2, based on the set of effective matching point pairs, selects the camera pair with the strongest geometric constraints as the initial camera pair by evaluating the baseline length, number and distribution of each camera pair; using the initial camera pair as a reference, the pose of the remaining cameras and the initial 3D point cloud are incrementally recovered using the triangulation method.

[0093] Specifically, all camera pairs with valid matching point pairs are traversed. For each camera pair, the number of valid matching point pairs exceeds a minimum threshold. If it does, the essential matrix or fundamental matrix between the two cameras is estimated using an eight-point or five-point method combined with the RANSAC algorithm, and the rotation matrix and normalized translation vector are decomposed from it. The magnitude of the translation vector reflects the relative length of the baseline; a baseline that is too long or too short is detrimental to the accuracy of 3D reconstruction. Simultaneously, the convex hull area of ​​the matching point pairs on the image's two-dimensional coordinates is calculated, or the proportion of non-empty grids in the grid distribution of matching point pairs is statistically analyzed to represent the uniformity of the matching point pair distribution. Considering the number of valid matching point pairs, the appropriateness of the estimated baseline length, and the uniformity of distribution, a comprehensive score is calculated using weighted summation and other methods. The camera pair with the highest score is selected as the initial camera pair for reconstruction.

[0094] The pose of the first camera in the initial camera pair is set as the identity matrix and zero vector, i.e., the origin of the world coordinate system. Based on the calculated relative pose, the pose of the second camera is determined. For the effective matching point pairs shared by the two cameras, their coordinates in 3D space are calculated using linear triangulation or nonlinear optimized triangulation, forming an initial 3D point cloud. From the unprocessed cameras, the camera that shares the most effective matching point pairs with the cameras whose poses have been recovered is selected as the next camera to be processed. Using the correspondence between the 2D feature points in this camera and the existing 3D point cloud, the pose of the new camera is solved using the perspective n-point method, i.e., the PnP algorithm combined with the RANSAC algorithm. After the pose is determined, this new camera is matched with other cameras whose poses have been recovered, and triangulation is performed on the newly observed matching points to increase the amount of 3D point cloud. This process is repeated until the poses of all cameras are recovered, forming a complete camera trajectory and a sparse 3D point cloud.

[0095] In an optional embodiment, the initial camera pair is selected as follows:

[0096] For any camera pair, the evaluation is performed by calculating the geometric constraint strength score of the camera pair, whereby the geometric constraint strength score is defined as... The calculation formula is as follows:

[0097] ;

[0098] In the formula, This represents the number of valid matching point pairs between two cameras in a camera pair. The disparity between the effective matching point pairs between the two cameras in a camera pair is calculated as the median pixel distance between the corresponding feature points in their respective image coordinate systems. The area of ​​the convex hull formed by the valid matching points on one of the images; , , These are the maximum values ​​of the corresponding indicators for all camera pairs; , , These are the preset weights for the corresponding indicators, and they satisfy... A larger parallax usually means that there is sufficient baseline length between cameras, which is beneficial for triangulation measurements. A larger convex hull area indicates a wider distribution of matching points and more stable geometric constraints.

[0099] S3, set the objective function as the weighted sum of the reprojection error term and the linear regularization term, determine the geometric principal axis of the current 3D point cloud, and calculate the weighted sum of the Euclidean distances from all 3D points in the 3D point cloud to the geometric principal axis as the linear regularization term; adjust the weight of the linear regularization term according to the average distance from all 3D points to the geometric principal axis, and periodically update the geometric principal axis during the optimization process; perform global bundle adjustment optimization to minimize the objective function.

[0100] Specifically, global bundle adjustment is a nonlinear least-squares optimization process. The objective function of global bundle adjustment consists of two parts. The first part is the standard reprojection error term, which is the sum of the squares of the pixel distances between the projected positions of all 3D points on all cameras that observe them and the actual observed positions of the 2D feature points. The second part is a linear regularization term. At the beginning of each optimization iteration or every fixed number of iterations, principal component analysis (PCA) is performed on the current set of all 3D point coordinates. The resulting first principal component vector defines the geometric principal axis of the 3D point cloud. This geometric principal axis is a straight line passing through the center of the point cloud, such as... Figure 2 As shown. The perpendicular distance from each 3D point to the geometric principal axis is calculated, and all perpendicular distances are summed to obtain the energy value of the linear regularization term. The weight coefficient of this linear regularization term is positively correlated with the average distance of all current 3D points to the geometric principal axis. The more severe the deviation of the 3D point cloud from the linear shape, the larger the average distance, the higher the weight, and the stronger the applied linear constraint. Optimization algorithms such as the Levenberg-Marquardt method are used to simultaneously adjust all camera pose parameters and all 3D point coordinates to minimize the objective function, thereby forcing the point cloud to tend towards a straight line while ensuring projection accuracy, thus correcting drift.

[0101] To identify the main extension direction of the 3D point cloud data, in an optional embodiment, the geometric principal axis is determined by principal component analysis, including:

[0102] Construct an M×3 coordinate matrix from the XYZ coordinates of all 3D points in the current 3D point cloud, where M is the number of 3D points;

[0103] Calculate the covariance matrix of the coordinate matrix;

[0104] The covariance matrix is ​​decomposed into eigenvalues ​​to obtain three eigenvalues ​​and three corresponding eigenvectors.

[0105] The eigenvector corresponding to the largest eigenvalue is selected as the direction vector of the geometric principal axis.

[0106] The coordinate matrix represents the spatial distribution of the point cloud; the covariance matrix reveals the dispersion of the point cloud data in each dimension and the correlation between dimensions. If the point cloud is stretched roughly along one direction, some terms of the covariance matrix will show strong correlation. Each eigenvector represents a direction in space, and the eigenvalue corresponding to the eigenvector represents the variance or distribution range of the point cloud in that direction. The direction corresponding to the largest eigenvalue is the direction in which the data changes most drastically.

[0107] To indicate the degree to which the current 3D point cloud deviates from its ideal linear structure, in an optional embodiment, the linear regularization term is calculated as follows:

[0108] Define the linear regularization term as The calculation formula is as follows:

[0109] in, For the first in a 3D point cloud One point, The total number of three-dimensional points. The geometric principal axis, For point To spindle The vertical Euclidean distance. The smaller the value, the more tightly the point cloud is arranged around the geometric principal axis, and the more obvious the linear structure of the scene. During the optimization process, minimizing the linear regularization term can encourage 3D points to move closer to the principal axis of the set, thereby correcting and straightening the scene curvature caused by camera pose drift.

[0110] In an optional embodiment, the method for adjusting the weights of the linear regularization term is as follows:

[0111] Before starting global bundle adjustment optimization, calculate the average Euclidean distance from all 3D points to the initial geometric principal axes, and use this average distance as the initial average distance. ;

[0112] Before each iteration of optimization, the average Euclidean distance from all 3D points to the current geometric principal axis is calculated and used as the current average distance. ;

[0113] The adjustment formula for the weight of the linear regularization term is: ;In the formula, The preset initial weighting coefficients, The weights are the linear regularization terms.

[0114] To ensure the geometric principal axis keeps pace with changes in the point cloud structure, it is recalculated every 20 iterations using the current point cloud position. This mechanism of adjusting weights and periodically updating the geometric principal axis allows linear constraints to have a mild effect in the early stages of optimization, while their effect increases as the point cloud structure stabilizes in the later stages, achieving intelligent and efficient scene correction.

[0115] S4. Project all source images onto a cylindrical canvas according to the optimized camera pose. Based on the gradient information entropy of each pixel on the cylindrical canvas in the overlapping area of ​​the source image projection, select the gradient information of the optimal source image to construct a global gradient field. Reconstruct a seamless 360° panoramic image by solving the Poisson equation.

[0116] A virtual cylinder is created based on the optimized camera trajectory, with its axis aligned with the geometric principal axis of the camera trajectory. Each source image is inversely projected onto this cylindrical canvas based on the optimized camera intrinsic and extrinsic parameters corresponding to the image. For any pixel on the cylindrical canvas, multiple source images may cover it. To determine the optimal information source for that pixel, for each source image covering the point, a small neighborhood window is taken at the corresponding original image location. The horizontal and vertical gradients of the image within this neighborhood window are calculated, and the information entropy is calculated based on the distribution of gradient values. Regions with higher information entropy generally indicate richer texture and clearer images. Therefore, the source image with the highest information entropy is selected, and its gradient value at that pixel is used as the target gradient value for that canvas pixel. This process is repeated for all pixels on the cylindrical canvas to obtain a complete global target gradient field. Using this gradient field as a known condition, the Poisson equation is efficiently solved using methods such as Fast Fourier Transform, and the panoramic image brightness value is reconstructed from the gradient field integral. It is ensured that the gradients at the stitching points of different source images are continuous, thus achieving visually seamless fusion.

[0117] To select the sharpest and most detailed image content for each pixel location in the panoramic image, in one optional embodiment, the global gradient field is constructed as follows:

[0118] For any pixel on the cylindrical canvas, find all the source images that can be projected onto that pixel; for each source image in the source image set, extract a neighborhood window of a preset size at the projection position of the pixel.

[0119] Calculate the gradient magnitude of the image within the neighborhood window, represent the gradient magnitude as 256 levels, and plot a histogram; calculate the gradient information entropy of the neighborhood window based on the histogram, and select the source image with the largest gradient information entropy as the optimal source image of the pixel.

[0120] Extract the horizontal and vertical gradient components of the optimal source image at the specified pixel to construct a global gradient field.

[0121] Specifically, let's take a pixel H on a cylindrical canvas as an example. Since multiple source images overlap, suppose there are three source images: image C, image D, and image E, all of which can cover the location of pixel H. To determine which image's content to use to fill point H, we examine the local texture information of each of these three images at their corresponding locations. We find the location corresponding to point H in image C and take a 5×5 pixel window centered on that location. We calculate the gradient of the 25 pixels within this window; the gradient reflects the edge and texture intensity of the image. We statistically analyze the calculated gradient values ​​to calculate the gradient entropy of the small window. The higher the entropy, the more complex the texture and the more details in that area. Let's assume the calculated entropy for the window corresponding to image C is 5.2, for image D it's 7.1, and for image E it's 4.5. By comparison, 7.1 is the highest value, indicating that image D detected the richest detail at point H, possibly because it was in the most accurate focus or had the highest resolution at that time. Therefore, it was decided to use the information from image D to define the gradient at point H. Specifically, this involves extracting the horizontal and vertical gradient values ​​of image D at point H. This process is repeated for every pixel on the cylindrical canvas, and all optimal gradient information is collected to form a complete global gradient field.

[0122] To generate a panoramic image with smooth color and brightness transitions using a global gradient field, in an optional embodiment, the process of reconstructing a seamless 360° panoramic image by solving the Poisson equation includes:

[0123] The divergence is calculated based on the constructed global gradient field;

[0124] Solve the Poisson equation by treating the divergence field as a right-hand side term. Where I is the intensity of the panoramic image to be solved, and G is the global gradient field. For the Laplace operator, For divergence operators;

[0125] Set Dirichlet boundary conditions to force the pixel intensity values ​​at the left and right edges of the cylindrical canvas to be equal;

[0126] The Poisson equation is solved using a solver based on Fast Fourier Transform to obtain a 360° panoramic image.

[0127] Specifically, the gradient field has a direction and magnitude at each pixel, while the divergence indicates whether that point is a source or sink of the gradient field, corresponding to the rate of change of brightness in the image. Calculating the divergence of the entire gradient field yields a divergence map, specifying the brightness variation characteristics that the image should have at each pixel. This divergence map is used as a known part of the Poisson equation, which aims to solve for a scalar field—the panoramic image—based on the given divergence information. This process can be understood as finding an image whose own gradient matches the global gradient field G. To ensure the panoramic image can be seamlessly stitched into a 360° loop, a boundary condition is applied, forcing the color values ​​of the leftmost column of pixels to be exactly the same as those of the rightmost column. An efficient numerical computation method, such as a Fast Fourier Transform solver, is used to solve the Poisson equation with the boundary condition. The result is the panoramic image, with natural color transitions, eliminating seams and exposure differences between different source images.

[0128] This invention filters effective matching point pairs by calculating the similarity of the spatial topology of the matching point pairs within their respective local neighborhoods. This accurately eliminates erroneous matches, significantly improves feature matching accuracy, and provides a reliable geometric correspondence for subsequent 3D reconstruction. First, this invention selects the initial camera pair with the strongest geometric constraints by evaluating the baseline length, the number and distribution of effective matching points. Then, it introduces a linear regularization term based on the geometric principal axis of principal component analysis in the global bundle adjustment. This regularization term, as a global geometric constraint, forces the point cloud and camera trajectory to conform to the linear physical shape of the subway vehicle, effectively suppressing drift and deformation caused by accumulated errors and ensuring the stability of 3D reconstruction. After projecting the source image onto a cylindrical canvas, this invention selects the source image gradient with the clearest texture for each pixel based on gradient information entropy, constructs a global gradient field, and then completes the fusion by solving the Poisson equation with Dirichlet boundary conditions. This preserves the details of the vehicle body while eliminating visual defects in overlapping areas, ultimately generating a high-precision, drift-free, and seamless 360° panoramic image of the subway vehicle, meeting the high requirements for panoramic image quality in scenarios such as rail transit operation and maintenance and appearance quality inspection.

[0129] Specific embodiments of the 360° image generation system for subway vehicles based on image stitching proposed in this invention are as follows:

[0130] A 360° image generation system for subway vehicles based on image stitching includes:

[0131] The filtering module is used to acquire source images collected along the subway vehicle, extract and match feature points between the source images to form an initial set of matching point pairs; calculate the spatial topological similarity of the two feature points of the matching point pairs in the initial set of matching point pairs in their respective local neighborhoods as the cohesion score, and filter the matching point pairs with scores greater than a preset threshold to form an effective set of matching point pairs.

[0132] The recovery module is used to select the camera pair with the strongest geometric constraints as the initial camera pair based on the set of effective matching point pairs by evaluating the baseline length, number and distribution of each camera pair; and incrementally recover the pose of the remaining cameras and the initial 3D point cloud using triangulation methods based on the initial camera pair.

[0133] The update module is used to set the objective function as a weighted sum of the reprojection error term and the linear regularization term, determine the geometric principal axes of the current 3D point cloud, and calculate the weighted sum of the Euclidean distances from all 3D points in the 3D point cloud to the geometric principal axes as the linear regularization term; adjust the weight of the linear regularization term according to the average distance from all 3D points to the geometric principal axes, and periodically update the geometric principal axes during the optimization process; perform global bundle adjustment optimization to minimize the objective function.

[0134] The reconstruction module projects all source images onto a cylindrical canvas according to the optimized camera pose. Based on the gradient entropy of each pixel on the cylindrical canvas in the overlapping area of ​​the source image projection, it selects the gradient information of the optimal source image to construct a global gradient field. By solving the Poisson equation, a seamless 360° panoramic image is reconstructed.

[0135] It should be noted that the processing procedures of the filtering module, the recovery module, the update module, and the reconstruction module are the same as those of steps S1, S2, S3, and S4 in the above embodiment of the method for generating 360° images of subway vehicles based on image stitching, and will not be described in detail here.

[0136] While various embodiments of the invention have been shown and described in this specification, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will occur to those skilled in the art without departing from the spirit and essence of the invention.

Claims

1. A 360° image generation method for a subway vehicle based on image stitching, characterized in that, The method comprises the following steps: obtaining source images collected along a subway vehicle, extracting and matching feature points between the source images to form an initial matching point pair set; calculating spatial topological graph structure similarity of two feature points of a matching point pair in the initial matching point pair set within respective local neighborhoods as a cohesion score, screening matching point pairs with scores greater than a preset threshold to form an effective matching point pair set; based on the effective matching point pair set, selecting a camera pair with the strongest geometric constraint as an initial camera pair by evaluating baseline length, effective matching point pair number and distribution of each camera pair; using a triangulation method to incrementally restore poses of the remaining cameras and an initial three-dimensional point cloud based on the initial camera pair as a reference; setting a target function as a weighted sum of a re-projection error term and a linear regularization term, determining a geometric principal axis of the current three-dimensional point cloud, and calculating a weighted sum of Euclidean distances of all three-dimensional points in the three-dimensional point cloud to the geometric principal axis as the linear regularization term; adjusting a weight of the linear regularization term according to an average distance of all three-dimensional points to the geometric principal axis, and periodically updating the geometric principal axis in the optimization process; performing global bundle adjustment optimization to minimize the target function; projecting all source images to a cylindrical canvas according to the optimized camera poses, selecting gradient information of an optimal source image to construct a global gradient field according to gradient information entropy of each pixel point in the source image projection overlapping region on the cylindrical canvas, and reconstructing a seamless 360° panoramic image by solving a Poisson equation; the method for constructing the effective matching point pair set comprises: for each feature point in a matching point pair, selecting a set number of feature points closest to the feature point in an image in which the feature point is located as a local neighborhood point set; and constructing a spatial topological graph with the feature point as the center and the normalized distance and angle between the feature point and the set number of feature points as edges; calculating a graph edit distance of two spatial topological graphs of the matching point pair to obtain a structure similarity score ranging from 0 to 1, and taking the structure similarity score as the cohesion score; setting a preset threshold of the cohesion score, retaining matching point pairs with scores greater than the preset threshold to form the effective matching point pair set. 2.The metro vehicle 360° image generation method based on image stitching according to claim 1, characterized in that, the method for selecting the initial camera pair comprises: For any camera pair, the evaluation is done by computing the geometric constraint strength score of this camera pair, defined as with the formula: ； In the formula, is the number of effective matching point pairs between two cameras in a camera pair, is the disparity of effective matching point pairs between two cameras in a camera pair, calculated as the median of the pixel distance of the corresponding feature points in the respective image coordinate systems; is the area of the convex hull formed by the effective matching points on one of the images; , , is the maximum value of the corresponding indicators among all camera pairs, respectively; , , is the preset weight of the corresponding indicators, respectively, and satisfies ; calculating geometric constraint strength scores of all camera pairs, and selecting a camera pair with the largest geometric constraint strength score as the initial camera pair. 3.The metro vehicle 360° image generation method based on image stitching according to claim 2, characterized in that, the method for incrementally restoring poses of the remaining cameras using the triangulation method comprises: selecting a camera that shares the most effective matching point pairs with a camera with a restored pose as a next camera to be processed from unprocessed cameras, and using a perspective n-point method combined with a RANSAC algorithm to solve the pose of the camera by using the correspondence between two-dimensional feature points in the camera and the existing three-dimensional point cloud. 4.The image stitching-based subway vehicle 360° image generation method according to claim 1, characterized in that, the geometric principal axis is determined by principal component analysis, comprising: constructing a coordinate matrix of all three-dimensional points in the current three-dimensional point cloud; calculating a covariance matrix of the coordinate matrix; performing eigenvalue decomposition on the covariance matrix to obtain three eigenvalues and corresponding three eigenvectors; selecting an eigenvector corresponding to the largest eigenvalue as a direction vector of the geometric principal axis. 5.The image stitching-based subway vehicle 360° image generation method according to claim 1, characterized in that, The adjustment method of the linear regularization term weight is: Before the global bundle adjustment optimization starts, compute the average of the Euclidean distances of all three-dimensional points to the initial geometric principal axes as the initial average distance ; Before each iteration of the optimization, the average of the Euclidean distances of all three-dimensional points to the current geometric principal axis is calculated as the current average distance ; The adjustment formula of the weight of the linear regularization term is: ; in the formula, is a preset initial weight coefficient, is the weight of the linear regularization term. 6.The image stitching-based subway vehicle 360° image generation method according to claim 1, characterized in that, The construction method of the global gradient field is: For any pixel point on the cylindrical canvas, find all source images that can be projected onto the pixel point; for each source image in the source image set, extract a neighborhood window of a preset size at the pixel point projection position; Calculate the gradient magnitude of the image in the neighborhood window, and express the gradient magnitude as 256 levels, and count the histogram; According to the histogram, calculate the gradient information entropy of the neighborhood window, and select the source image with the maximum gradient information entropy as the optimal source image of the pixel point; Extract the horizontal gradient component and the vertical gradient component of the optimal source image at the pixel point, and construct a global gradient field.

7. The subway vehicle 360° image generation method based on image stitching according to claim 6, characterized in that, The method for reconstructing a seamless 360° panoramic image by solving a Poisson equation is: Calculate the divergence according to the constructed global gradient field; Solve the Poisson equation with the divergence field as the right side item of the Poisson equation ; wherein I is a panoramic image intensity to be solved, G is a global gradient field, is a Laplace operator, is a divergence operator; Set the Dirichlet boundary condition to force the pixel intensity values of the left and right ends of the cylindrical canvas to be equal; Solve the Poisson equation using a solver based on fast Fourier transform to obtain a 360° panoramic image.

8. A 360° image generation system for a metro vehicle based on image stitching, characterized in that, It comprises: A screening module is configured to acquire source images collected along a metro vehicle, extract and match feature points between the source images, and form an initial matching point pair set; The similarity of the spatial topology structure of the two feature points of the matching point pair in the initial matching point pair set in their respective local neighborhoods is calculated as the cohesion score, and the matching point pairs with a score greater than a preset threshold are screened to form an effective matching point pair set; A recovery module is configured to select the camera pair with the strongest geometric constraint as an initial camera pair based on the effective matching point pair set by evaluating the baseline length, the number and distribution of effective matching point pairs of each camera pair; and incrementally recover the poses of the remaining cameras and the initial three-dimensional point cloud using a triangulation method with the initial camera pair as a reference; An updating module is configured to set a target function as a weighted sum of a re-projection error term and a linear regularization term, determine a geometric principal axis of the current three-dimensional point cloud, and calculate a weighted sum of the Euclidean distances of all three-dimensional points in the three-dimensional point cloud to the geometric principal axis as the linear regularization term; The weight of the linear regularization term is adjusted according to the average distance of all three-dimensional points to the geometric principal axis, and the geometric principal axis is periodically updated during the optimization process; Perform global bundle adjustment optimization to minimize the target function; A reconstruction module is configured to project all source images onto a cylindrical canvas according to the optimized camera poses, select the gradient information of the optimal source image to construct a global gradient field according to the gradient information entropy of each pixel point on the cylindrical canvas in the overlapping region of the source images, and reconstruct a seamless 360° panoramic image by solving a Poisson equation. The method for constructing the set of effective matching point pairs comprises: for each feature point in each matching point pair, selecting a set of a predetermined number of feature points closest to the feature point in the image in which the feature point is located as a local neighborhood point set; constructing a spatial topology graph with the feature point as the center and with the normalized distance and the included angle between the feature point and the predetermined number of feature points as edges; obtaining a structure similarity score ranging from 0 to 1 by calculating the graph edit distance of the two spatial topology graphs of the matching point pair, and taking the structure similarity score as a cohesion score; setting a preset threshold of the cohesion score, retaining the matching point pairs with scores greater than the preset threshold, and constructing the set of effective matching point pairs.

Images