Multi-scale linear array camera splicing method and system based on point cloud

Abstract

The invention discloses a multi-scale linear array camera splicing method and system based on point cloud. The method comprises the following steps: completing acquisition and preprocessing of point cloud data and image data of a target area; determining an overlapping region range between adjacent images; extracting spatial structure characteristics in the point cloud data, and performing multi-scale hierarchical decomposition on the point cloud through a multi-scale segmentation method; meanwhile, multi-scale image feature extraction is carried out on the images of the linear array camera; solving gradients in X and Y directions by adopting an optical flow method aiming at any pixel in the overlapping region, and calculating a motion vector between the pixels; fusing the optical flow information obtained under each scale, and constructing a globally consistent optical flow vector field; according to the fused optical flow vector, calculating to obtain a geometric transformation matrix of the whole overlapping region; and after image transformation and alignment are completed through the transformation matrix, fusion processing is carried out on overlapped areas. And the unification of the visual effect and the spatial integrity of the spliced image is ensured.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of line array camera image stitching, and more particularly relates to a multi-scale line array camera stitching method and system based on point clouds. BACKGROUND

[0002] Line array cameras are widely used in image acquisition in the fields of industrial detection, remote sensing and mapping due to their high resolution and wide dynamic range. However, the field of view of a single line array camera is limited, and a large scene image needs to be acquired through image stitching. Traditional image stitching relies on feature matching algorithms, such as SIFT and SURF-based feature point matching, and the transformation matrix is calculated by finding the same name feature points to realize the stitching. However, this method has few feature points in scenes with lack of texture, low matching accuracy, and is sensitive to changes in light and angle, which easily causes misalignment and unnatural transition in the overlapping area.

[0003] With the development of three-dimensional data acquisition technology, the point cloud-based stitching method uses three-dimensional information of point clouds to assist image alignment, which improves the limitations of traditional methods. However, existing technologies mostly use single-scale point clouds, which are easily affected by noise at small scales and lose local details at large scales, and cannot balance different scale features and details, making it difficult to achieve high-precision stitching in complex scenes.

[0004] In addition, traditional optical flow algorithms are based on single-scale image information in the processing of overlapping areas, which is difficult to fully capture the motion information of targets of different sizes, resulting in deviation of motion vectors, affecting the accuracy of geometric transformation matrix, and causing misalignment or deformation of the stitched image. At the same time, existing stitching methods lack sufficient processing of the seam and brightness difference in the overlapping area during image fusion, resulting in unnatural visual transition of the stitched image. Moreover, there is a lack of global optimization mechanism, making it difficult to use three-dimensional information of point clouds to correct optical flow estimation errors, resulting in inaccurate overall geometric relationship of the stitched image, especially in complex environments, making it difficult to guarantee the stitching accuracy and robustness.

[0005] Therefore, how to use the multi-scale characteristics of point clouds to accurately calculate the geometric transformation relationship of images combined with optical flow algorithms, improve the fusion quality of the overlapping area, and realize high-precision and robust line array camera image stitching is a problem to be solved. The present application aims to solve the above problems through multi-scale point cloud and image feature processing, optical flow fusion, geometric transformation matrix calculation and optimization, etc., to improve the stitching quality and reliability. SUMMARY

[0006] The present application aims to solve the problems existing in the prior art of line array camera image stitching technology. In view of the low matching rate of traditional methods in texture-poor scenes, the insufficient precision of single-scale processing based on point clouds, the deviation of motion vector calculation of optical flow algorithm, the poor fusion effect of overlapping areas, and the lack of global optimization, through multi-scale point cloud and image feature collaborative processing, combined with optical flow fusion and geometric transformation matrix accurate calculation, the overlapping area fusion is optimized, and high-precision and robust line array camera image stitching is realized.

[0007] In view of the above defects or improvement needs of the prior art, as a first aspect of the present application, the present application provides a multi-scale point cloud-based line array camera stitching method, comprising: S1. Collect and preprocess the point cloud data and image data of the target area; S2. Determine the overlapping area range between adjacent images through point cloud data and camera calibration information; extract spatial structure features in the point cloud data, and perform multi-scale hierarchical decomposition of the point cloud through a multi-scale segmentation method; correspondingly, perform multi-scale image feature extraction on the line array camera image, and capture edge, texture and structure information at different scales; S3. For any pixel in the overlapping area, use the optical flow method to solve the X direction gradient and the Y direction gradient, and calculate the motion vector between the pixels; and fuse the optical flow information obtained at each scale to construct a globally consistent optical flow vector field; according to the fused optical flow vector, calculate the geometric transformation matrix of the entire overlapping area; S4. After image transformation and alignment through the transformation matrix, perform fusion processing on the overlapping area.

[0008] Further, the preprocessing process in S1 is specifically: The collected point cloud and image data are preprocessed through an internal and external parameter correction method to realize the alignment of their spatial coordinates and time; and the continuous image data collected by the line array camera are preprocessed including noise suppression, white balance adjustment and brightness normalization.

[0009] Further, the specific method for determining the overlapping area range between adjacent images in S2 is: Project the point cloud to the adjacent image plane through the camera calibration parameters, specifically: , , In the formula, is the coordinate of the three-dimensional point in the world coordinate system; is the coordinate of the three-dimensional point in the camera coordinate system; is the rotation matrix, is the three-dimensional translation vector; is unwound components; is the camera intrinsic matrix; is the focal length; is the principal point coordinate, which is the pixel coordinate of the intersection of the camera optical axis and the image plane; is the coordinate of the point in the image pixel coordinate system; According to the projection point density, the overlapping region is determined, and finally the feature matching verification is completed to determine the range of the adjacent image overlapping region.

[0010] Further, the specific method for determining the overlapping region according to the projection point density is: The projection point density in the neighborhood of the pixel point on the image plane is defined as: wherein, is the pixel coordinate of the i-th projection point in the neighborhood ; is the neighborhood radius; is the Gaussian kernel parameter; and the high-density region is screened through a threshold wherein, is the image domain, and the candidate overlapping region.

[0011] Further, the specific method for the feature matching verification is: The feature matching score of the points in the candidate region is calculated: wherein, is the region centered on the pixel , and the comprehensive score in the feature matching; is the pixel point of the current image to be calculated for the matching score; is the corresponding pixel point in the adjacent image; is the pre-screened candidate matching region in the adjacent image; is a spatial weight function; is a feature descriptor, and is a feature similarity measurement function; Combining the point cloud projection density and the feature matching score , a comprehensive confidence is calculated: wherein,​​​​​​​​​​ is a weight coefficient; and a confidence threshold is set determining the final overlapping region: , wherein, represents the final determined overlapping region.

[0012] Further, the multi-scale segmentation method in S2 is specifically: Given the original point cloud wherein, is a three-dimensional space point; Set multi-scale parameters , construct a voxel grid : , wherein, is the feature value indexed by two-dimensional discrete coordinates at the th scale, and the center point coordinates are ; represents an integer set; Map each point in the original point cloud to the corresponding voxel according to the formula , and record the voxel containing the point set ; calculate the barycenter of each point in the voxel, and generate the sparse point cloud : , , wherein, represents the number of points in the voxel; sequentially generate the point cloud at the th scale, wherein , the larger the scale, the sparser the point cloud.

[0013] Further, the specific calculation method of the geometric transformation matrix in S3 is: For the overlapping region pixels, fuse the multi-scale optical flow vectors using the equal weight average method: , wherein, is the final output average optical flow vector; represents the total number of scales; represents the optical flow vector at the th scale; uniformly select Each feature point records its original coordinates. Coordinates of optical flow target ,in: , , According to the affine transformation method, Substituting the coordinates of the feature points, we construct an overdetermined system of linear equations: , In the formula, These are the parameters for linear transformation; The parameters are for translation transformation; the parameters are solved directly using the least squares method. : , in, , The feature point coordinate matrix The target coordinate matrix is ​​obtained; the final radial transformation matrix is ​​obtained. : , Then calculate the average error between the transformed feature points and the optical flow target coordinates. ,when When the threshold is exceeded, manually remove the errors with the largest errors. For each point, recalculate the affine matrix.

[0014] Furthermore, the average error The calculation method is as follows: , in, These are the transformed coordinates.

[0015] As a second aspect of the present invention, a multi-scale linear array camera stitching system based on point clouds is also provided, comprising: The data acquisition and preprocessing unit is used to acquire and preprocess point cloud data and image data of the target area. The multi-scale feature extraction unit is used to determine the overlapping area between adjacent images through point cloud data and camera calibration information; and to extract spatial structure features from the point cloud data, and to decompose the point cloud into multi-scale hierarchical components through a multi-scale segmentation method; correspondingly, multi-scale image feature extraction is performed on the linear array camera image to capture edge, texture and structural information at different scales. An optical flow vector calculation unit is configured to solve X direction gradient and Y direction gradient by using an optical flow method for any pixel in the overlapping area, and to calculate the motion vector between pixels; and to fuse the optical flow information obtained at each scale to construct a globally consistent optical flow vector field; and to calculate the geometric transformation matrix of the whole overlapping area according to the fused optical flow vector. An image transformation fusion unit is configured to perform fusion processing on the overlapping area after image transformation and alignment are completed by using the transformation matrix.

[0016] As a third aspect of the present application, a computer readable storage medium is provided, which stores a computer program for implementing any step of the point cloud based multi-scale linear array camera stitching method.

[0017] Overall, compared with the prior art, the above technical solutions conceived by the present application can achieve the following beneficial effects: 1. The point cloud based multi-scale linear array camera stitching method of the present application determines the range of the overlapping area between adjacent images by using point cloud data and camera calibration information, and performs multi-scale hierarchical decomposition on the collected point cloud to extract its spatial structure features, so that the spatial distribution of the point cloud at different scales can be presented; correspondingly, multi-scale feature extraction is carried out on the images obtained by the linear array camera to capture edge, texture and structure information at each scale. This process fully excavates the multi-scale features of the point cloud and the images, and builds a cross-modal and multi-scale feature correlation system, which lays a multi-dimensional feature foundation for the alignment and fusion of images in the subsequent stitching process, and guarantees the effective use and matching of features at different scale dimensions.

[0018] 2. The point cloud based multi-scale linear array camera stitching method of the present application solves X direction gradient and Y direction gradient by using an optical flow method for any pixel in the overlapping area, calculates the motion vector between pixels, and then fuses the optical flow information obtained at each scale to construct a globally consistent optical flow vector field. Based on the fused optical flow vector, the geometric transformation matrix of the whole overlapping area is derived and calculated. The optical flow method captures the motion trend of the pixels, and the multi-scale optical flow fusion makes the vector field reflect the global motion law. The geometric transformation matrix converts the pixel motion relationship reflected by the optical flow into a mathematical model of image space transformation, provides a calculation basis for image alignment, realizes the connection from local pixel motion analysis to overall image space transformation modeling, solves the image misplacement problem caused by factors such as shooting angle and motion, and provides mathematical support for subsequent stitching.

[0019] 3. The point cloud-based multi-scale linear array camera stitching method of the present application, through the calculated geometric transformation matrix, completes the transformation and alignment of the images, and then performs fusion processing on the overlapping area. The geometric transformation matrix drives the spatial position adjustment of the image, aligns the adjacent images in the overlapping area, and reduces the stitching displacement caused by the positional deviation; the overlapping area fusion link, based on the alignment result, smoothes the color and texture differences between the images, and reduces the stitching gap. This process takes multi-scale feature processing, optical flow field construction and matrix calculation as the premise, forms a complete closed loop of "feature extraction - transformation modeling - alignment fusion", effectively handles the image misplacement and obvious stitching marks in linear array camera image stitching, outputs a coherent and complete stitched image, realizes the goal of linear array camera multi-image stitching, and ensures the unity of the visual effect and spatial integrity of the stitched image. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 The point cloud-based multi-scale linear array camera stitching method flowchart of the present application embodiment; Figure 2 The stitched image schematic diagram of the present application embodiment; Figure 3 The system unit diagram of the present application embodiment. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0022] Embodiment 1 Please refer to Figure 1 The present embodiment 1 provides a point cloud-based multi-scale linear array camera stitching method, comprising: S1. Complete the acquisition and preprocessing of the target area point cloud data and image data; S2. Determine the overlapping area range between adjacent images through point cloud data and camera calibration information; and extract spatial structure features in the point cloud data, and perform multi-scale hierarchical decomposition of the point cloud through a multi-scale segmentation method; correspondingly, perform multi-scale image feature extraction on the linear array camera image, and capture edge, texture and structure information at different scales; S3. For the overlapping region, any pixel adopts the optical flow method to solve the X direction gradient and Y direction gradient, and calculate the motion vector between pixels; and the optical flow information obtained at each scale is fused to construct a globally consistent optical flow vector field; according to the fused optical flow vector, the geometric transformation matrix of the whole overlapping region is calculated; S4. After the image transformation and alignment are completed by the transformation matrix, the overlapping region is fused.

[0023] This embodiment 1 further expands the above steps.

[0024] (1) Data acquisition and preprocessing This embodiment 1 obtains target region point cloud data and image data through synchronous acquisition and preprocessing. Laser scanning or structured light sensor is used to collect point cloud data, recording the three-dimensional space coordinates of the object surface; line array camera is used to collect image data in a continuous shooting manner, obtaining the texture and color information of the target region.

[0025] After data acquisition, preprocessing is performed by using internal and external parameter correction method. For image data, noise is removed by filtering method, color deviation is adjusted by white balance, and brightness normalization processing is performed to unify the brightness difference between images, providing stable texture information for optical flow calculation. For point cloud data, the internal parameters (focal length, principal point position, etc.) and external parameters (camera space position and attitude) of the camera are accurately calibrated, the spatial mapping relationship between point cloud and image is established, and a three-dimensional geometric model is constructed, providing a spatial reference for determining the image overlapping region and image correction.

[0026] Through the above preprocessing, the point cloud data and image data are aligned in space coordinates and time, the data deviation is eliminated, and the data quality is improved, providing a reliable data basis for subsequent operations such as determining the overlapping region range, extracting multi-scale features, calculating optical flow and geometric transformation matrix, etc.

[0027] (2) Multi-scale feature extraction This embodiment 1 determines the overlapping region range of adjacent images by using point cloud data and camera calibration information, and extracts multi-scale features. The point cloud is projected to the adjacent image plane through the camera calibration parameters, the candidate overlapping region is preliminarily screened according to the projection point density, and the comprehensive confidence is calculated through feature matching verification, and finally the overlapping region range is determined, which limits the area for subsequent optical flow calculation.

[0028] In a preferred embodiment, the specific method for determining the overlapping region range between adjacent images is as follows: The point cloud is projected to the adjacent image plane through the camera calibration parameters, specifically as follows: , , In the formula, is the coordinate of the three-dimensional point in the world coordinate system; is the coordinate of the three-dimensional point in the camera coordinate system; is a rotation matrix, is a three-dimensional translation vector; is the decomposed components; is an intrinsic matrix of the camera; is a focal length; is a principal point coordinate, which is the pixel coordinate of the intersection of the camera optical axis and the image plane; is the coordinate of the point in the image pixel coordinate system; According to the projection point density, the overlapping region is determined, and finally the feature matching verification is completed to determine the range of the overlapping region of the adjacent images.

[0029] In a preferred embodiment, the specific method for determining the overlapping region according to the projection point density is as follows: The projection point density in the neighborhood of the pixel point on the image plane is defined as: , wherein, is the pixel coordinate of the i-th projection point falling in the neighborhood is a neighborhood radius; is a Gaussian kernel parameter; and the high-density region is screened through a threshold , wherein, is the image domain, is a candidate overlapping region.

[0030] In a preferred embodiment, the specific method for feature matching verification is as follows: The feature matching score of the points in the candidate region is calculated: , wherein, is a region centered on the pixel is the comprehensive score in feature matching; is the pixel point in the current image to be calculated for the matching score; is the corresponding pixel point in the adjacent image; is the pre-screened candidate matching region in the adjacent image; is a spatial weight function; ​​​​​​​This is a feature similarity measurement function; Combined point cloud projection density Feature matching score Calculate the overall confidence level : , In the formula, These are the weighting coefficients; then, they are passed through a confidence threshold. Determine the final overlapping area: , In the formula, This indicates the final determined overlapping area.

[0031] In terms of multi-scale segmentation of point clouds, given the original 3D point cloud, a voxel mesh is constructed by setting multi-scale parameters. Each point is mapped to the corresponding voxel according to the scale parameters. The centroid of the points in each voxel is calculated to generate sparse point clouds of different scales. The larger the scale, the sparser the point cloud, thus realizing multi-scale hierarchical decomposition of point clouds and extracting spatial structural features.

[0032] In a preferred embodiment, the multi-scale segmentation method specifically includes: Given the original point cloud ,in A point in three-dimensional space; Setting multi-scale parameters Constructing a voxel mesh : , in, For the first At each scale, using two-dimensional discrete coordinates The eigenvalues ​​of the index, whose center point coordinates are... ; Represents the set of integers; Original point cloud Each point in According to the formula Map to the corresponding voxel, and record the voxel. The set of points included is Calculate the centroid for each point within each voxel. Generate a sparsed point cloud : , , In the formula, Indicates the number of points within a voxel; generated sequentially. Point clouds at various scales ,in The larger the scale, the sparser the point cloud.

[0033] For the line array camera image, a multi-scale pyramid is constructed, starting from the original image, generating multi-layer images by high-pass filtering and down-sampling, finding extreme points by Gaussian difference, dividing regions around the key points to calculate gradient histograms to generate descriptors, and screening key points by the FAST method, comparing the gray levels of pixels in the neighborhood of the key points to generate binary descriptors, realizing multi-scale image feature extraction, and capturing edge, texture and structure information at different scales.

[0034] Further, for each scale ( ), the point cloud is associated with the corresponding image pyramid level ; using the camera intrinsic parameters and extrinsic parameters , the points are projected to the image plane: , wherein is the pixel coordinate; in the image level , the image features near the point are extracted and associated with the point cloud features (such as density, height difference).

[0035] (3) Optical flow vector calculation This embodiment 1 realizes the analysis of the motion of the pixels in the overlapping area and the modeling of the image transformation through multi-scale optical flow estimation and geometric transformation matrix calculation. In the multi-scale optical flow estimation link, for the pixels in the overlapping area, the optical flow method is used to solve the X and Y direction gradients in each level of the image pyramid, and the motion vectors between pixels are calculated. Specifically, the coarse optical flow field is calculated from the coarsest level, which is up-sampled to the next detailed layer as the initial value for optimization, and is refined layer by layer, and finally the pixel-level optical flow field at full resolution is obtained, which reflects the corresponding relationship of any pixel points in adjacent images. Then, the optical flow information at each scale is fused to construct a globally consistent optical flow vector field, which is used to reflect the displacement of the pixels in the overlapping area.

[0036] In the calculation of the geometric transformation matrix, the multi-scale optical flow vectors of the pixels in the overlapping area are fused by equal weight averaging method to obtain the average optical flow vector. N feature points are uniformly selected in the overlapping area, and their original coordinates and target coordinates calculated according to the average optical flow vector are recorded. According to the principle of affine transformation, these feature point coordinates are substituted into the constructed overdetermined linear equation system, and the affine transformation parameters are solved by the least square method to obtain the affine transformation matrix. The average error between the transformed feature points and the target coordinates of the optical flow is calculated, and if the error exceeds the threshold, the K points with the largest error are manually removed, and the affine matrix is recalculated until the requirement is met.

[0037] In a preferred embodiment, the specific calculation method of the geometric transformation matrix is: Fusing multi-scale optical flow vectors for overlapping region pixels An equal-weight average method is used: , wherein, is the average optical flow vector of the final output; represents the total number of scales; represents the optical flow vector at the th scale; In the overlapping region, evenly select feature points, record the original coordinates and the optical flow target coordinates , wherein: , , According to the affine transformation method, substitute feature point coordinates to construct an overdetermined linear equation set: , wherein, is the linear transformation parameter; is the translation transformation parameter; use the least square method to directly solve the parameters : , wherein, , is the feature point coordinate matrix, is the target coordinate matrix; finally obtain the radiation transformation matrix : , Then calculate the average error of the transformed feature points and the optical flow target coordinates When exceeds the threshold value; by manually removing the largest error points, recalculate the affine matrix.

[0038] In a preferred embodiment, the calculation method of the average error is: , wherein, is the transformed coordinate.

[0039] (4) Image transformation fusion Please refer to Figure 2After the completion of the geometric transformation matrix calculation, the embodiment 1 enters the image transformation and alignment and fusion processing link. The target image is subjected to geometric transformation by using the calculated transformation matrix, the pixel coordinates of the target image are mapped to the positions corresponding to the overlapping area of the reference image through matrix operation, the spatial positions of adjacent images in the overlapping area are aligned, and the pixel positions of the same scene content in the two images are accurately corresponded.

[0040] For the overlapping area, a weighted average, Poisson fusion and other image fusion methods are used for processing. The weighted average assigns weights according to the distance of the pixel to the boundary of the overlapping area, so that the edge pixels are smoothly transitioned; the Poisson fusion realizes natural connection of texture by calculating the gradient field, effectively eliminates the splicing joint and brightness difference, makes the image color and texture of the overlapping area transition naturally, and obtains a visually smooth and coherent splicing effect.

[0041] If there is three-dimensional point cloud data, the preliminary splicing result is combined with the three-dimensional model constructed by the point cloud, and the error is further corrected through global geometric optimization. The point cloud data provides the three-dimensional spatial structure information of the scene, associates the pixel points in the spliced image with the point cloud coordinates in the three-dimensional model, adjusts the image transformation parameters through the optimization method, corrects the slight misplacement caused by the optical flow estimation error, ensures that the overall geometric relationship of the spliced image is consistent with the actual scene, and improves the spatial accuracy of splicing.

[0042] The multi-scale features after fusion are used to automatically generate a global splicing model. The multi-scale features include image details and point cloud structure information at different resolutions, and the model models the geometric relationship of the entire spliced image based on these features, identifies and locates the local splicing error area, compensates and optimizes the error through parameter adjustment, and further improves the overall accuracy of the spliced image.

[0043] In addition, an online feedback and iterative updating strategy is designed. In actual application, the system continuously collects new point cloud and image data, compares and analyzes with the spliced result, detects the change of splicing error. According to the feedback information, the parameters and model of the splicing method are iteratively updated, which adapts to environmental changes, ensures that in complex scenes such as light mutation and object motion, the splicing accuracy and robustness can still be maintained, and makes the splicing system have the ability of continuous optimization.

[0044] Embodiment 2 Please refer to Figure 3 The embodiment 2 provides a multi-scale line array camera splicing system based on point cloud, which comprises: A data acquisition and preprocessing unit is configured to acquire and preprocess point cloud data and image data of a target area. The multi-scale feature extraction unit is configured to determine the range of the overlapping area between adjacent images by using the point cloud data and the camera calibration information, extract spatial structure features in the point cloud data, and perform multi-scale hierarchical decomposition on the point cloud by using a multi-scale segmentation method; correspondingly, multi-scale image feature extraction is performed on the linear array camera image to capture edge, texture and structure information at different scales; The optical flow vector calculation unit is configured to solve the X-direction gradient and the Y-direction gradient by using an optical flow method for any pixel in the overlapping area, and calculate the motion vector between the pixels; the optical flow information obtained at different scales is fused to construct a globally consistent optical flow vector field; and the geometric transformation matrix of the whole overlapping area is calculated according to the fused optical flow vector. The image transformation fusion unit is configured to perform fusion processing on the overlapping area after image transformation and alignment are completed by using the transformation matrix.

[0045] Embodiment 3 The embodiment 3 also provides a computer readable storage medium, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement any step of the multi-scale linear array camera splicing method based on point cloud.

[0046] The computer readable storage medium can include a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0047] For the computer readable storage medium provided in the present application, refer to the above method embodiments, and the present application will not be repeated here.

[0048] Those skilled in the art can easily understand that the above description is only a preferred embodiment of the present application, and is not used to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A multi-scale linear array camera stitching method based on point clouds, characterized in that, include: S1. Complete the acquisition and preprocessing of point cloud data and image data of the target area; S2. Determine the overlapping area between adjacent images using point cloud data and camera calibration information; extract spatial structure features from the point cloud data, and decompose the point cloud into multi-scale hierarchical components using a multi-scale segmentation method; correspondingly, extract multi-scale image features from the line scan camera images to capture edge, texture, and structural information at different scales. S3. For any pixel in the overlapping region, use the optical flow method to solve the gradient in the X and Y directions, and calculate the motion vector between pixels; then fuse the optical flow information obtained at each scale to construct a globally consistent optical flow vector field; and calculate the overall geometric transformation matrix of the overlapping region based on the fused optical flow vector. S4. After image transformation and alignment are completed using the transformation matrix, the overlapping areas are fused.

2. The multi-scale linear array camera stitching method based on point clouds according to claim 1, characterized in that, The preprocessing process in S1 is specifically as follows: The acquired point cloud and image data are preprocessed using intrinsic and extrinsic parameter correction methods to achieve spatial and temporal alignment between the two; and the continuous image data acquired by the line scan camera is preprocessed including noise suppression, white balance adjustment and brightness normalization.

3. The multi-scale linear array camera stitching method based on point clouds according to claim 1, characterized in that, The specific method for determining the overlapping region range between adjacent images in S2 is as follows: The point cloud is projected onto an adjacent image plane using camera calibration parameters, specifically as follows: ， ， In the formula, These are the coordinates of a 3D point in the world coordinate system. These are the coordinates of a 3D point in the camera coordinate system. For rotation matrix, It is a three-dimensional translation vector; for The expanded components; This is the camera intrinsic parameter matrix; Focal length; Principal point coordinates are the pixel coordinates of the intersection of the camera's optical axis and the image plane. The coordinates of the point in the image pixel coordinate system; Then, the overlapping area is determined based on the density of the projection points, and finally, the range of the overlapping area between adjacent images can be determined by completing the feature matching verification.

4. The multi-scale linear array camera stitching method based on point clouds according to claim 3, characterized in that, The specific method for determining the overlapping region based on the projection point density is as follows: Define pixels on the image plane neighborhood Projection point density within for: ， in, To fall in the neighboring area The first in The pixel coordinates of each projection point; The neighborhood radius; The parameters are Gaussian kernel parameters; then a threshold is applied. Filter high-density areas: ， in, For the image domain, These are candidate overlapping regions.

5. The multi-scale linear array camera stitching method based on point clouds according to claim 3, characterized in that, The specific method for feature matching verification is as follows: Calculate candidate regions Interior point feature matching score: ， in, In pixels The overall score for feature matching in the central region; The pixels in the current image for which matching scores are to be calculated; For adjacent images with The corresponding pixels; These are pre-selected candidate matching regions in adjacent images; This is the spatial weighting function; For feature descriptors, This is a feature similarity measurement function; Combined point cloud projection density Feature matching score Calculate the overall confidence level : ， In the formula, These are the weighting coefficients; then, they are passed through a confidence threshold. Determine the final overlapping area: ， In the formula, This indicates the final determined overlapping area.

6. The multi-scale linear array camera stitching method based on point clouds according to claim 1, characterized in that, The multi-scale segmentation method in S2 is specifically as follows: Given the original point cloud ,in A point in three-dimensional space; Setting multi-scale parameters Constructing a voxel mesh : ， in, For the first At each scale, using two-dimensional discrete coordinates The feature value of the index, whose center point coordinates are ; Represents the set of integers; Original point cloud Each point in According to the formula Map to the corresponding voxel, and record the voxel. The set of points included is Calculate the centroid for each point within each voxel. Generate a sparsed point cloud : ， ， In the formula, Indicates the number of points within a voxel; generated sequentially. Point clouds at various scales ,in The larger the scale, the sparser the point cloud.

7. The multi-scale linear array camera stitching method based on point clouds according to claim 1, characterized in that, The specific calculation method for the geometric transformation matrix in S3 is as follows: For pixels in overlapping regions, fuse multi-scale optical flow vectors. The equal-weighted average method is used: ， In the formula, This is the final output average optical flow vector; Indicates the total number of scales; Indicates the first Optical flow vectors at various scales; Select uniformly in the overlapping region Each feature point records its original coordinates. Coordinates of optical flow target ,in: ， ， According to the affine transformation method, Substituting the coordinates of the feature points, we construct an overdetermined system of linear equations: ， In the formula, These are the parameters for linear transformation; The parameters are for translation transformation; the parameters are solved directly using the least squares method. : ， in, , The feature point coordinate matrix The target coordinate matrix is ​​obtained; the final radial transformation matrix is ​​obtained. : ， Then calculate the average error between the transformed feature points and the optical flow target coordinates. ,when When the threshold is exceeded, manually remove the errors with the largest errors. For each point, recalculate the affine matrix.

8. The multi-scale linear array camera stitching method based on point clouds according to claim 7, characterized in that, The average error The calculation method is as follows: ， in, These are the transformed coordinates.

9. A multi-scale linear array camera stitching system based on point clouds, characterized in that, include: The data acquisition and preprocessing unit is used to acquire and preprocess point cloud data and image data of the target area. The multi-scale feature extraction unit is used to determine the overlapping area between adjacent images through point cloud data and camera calibration information; and to extract spatial structure features from the point cloud data, and to decompose the point cloud into multi-scale hierarchical components through a multi-scale segmentation method; correspondingly, multi-scale image feature extraction is performed on the linear array camera image to capture edge, texture and structural information at different scales. The optical flow vector calculation unit is used to solve the X-direction gradient and Y-direction gradient for any pixel in the overlapping region using the optical flow method, and to calculate the motion vector between pixels; it also fuses the optical flow information obtained at each scale to construct a globally consistent optical flow vector field; and calculates the overall geometric transformation matrix of the overlapping region based on the fused optical flow vector. The image transformation and fusion unit is used to perform image transformation and alignment using a transformation matrix, and then fuse the overlapping areas.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is executed by a processor according to any one of claims 1-8, which describes a multi-scale linear array camera stitching method based on point clouds.

Images