A point cloud-based multi-scale linear array camera stitching method and system

Abstract

The application discloses a kind of multi-scale line array camera splicing method and system based on point cloud, method includes: the acquisition and pre-processing of target area point cloud data and image data are completed;Determine the range of overlapping area between adjacent images;And extract spatial structure features in point cloud data, carry out multi-scale hierarchical decomposition to point cloud by multi-scale segmentation method;While multi-scale image feature extraction is carried out to line array camera image;For any pixel in the overlapping area, solve X, Y direction gradient by using optical flow method, and calculate the motion vector between pixels;And the optical flow information obtained under each scale is fused, to construct global consistent optical flow vector field;According to the fused optical flow vector, the geometric transformation matrix of the whole overlapping area is calculated;After image transformation and alignment are completed by transformation matrix, the overlapping area is fused. Guarantee that the image after splicing is uniform in visual effect and spatial integrity.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of line array camera image stitching, and more particularly relates to a multi-scale line array camera stitching method and system based on point clouds. BACKGROUND

[0002] Line array cameras are widely used in image acquisition in the fields of industrial detection, remote sensing and mapping due to their high resolution and wide dynamic range. However, the field of view of a single line array camera is limited, and a large scene image needs to be acquired through image stitching. Traditional image stitching relies on feature matching algorithms, such as SIFT and SURF-based feature point matching, and the transformation matrix is calculated by finding the same name feature points to realize the stitching. However, this method has few feature points in scenes with lack of texture, low matching accuracy, and is sensitive to changes in light and angle, which easily causes misalignment and unnatural transition in the overlapping area.

[0003] With the development of three-dimensional data acquisition technology, the point cloud-based stitching method uses three-dimensional information of point clouds to assist image alignment, which improves the limitations of traditional methods. However, existing technologies mostly use single-scale point clouds, which are easily affected by noise at small scales and lose local details at large scales, and cannot balance different scale features and details, making it difficult to achieve high-precision stitching in complex scenes.

[0004] In addition, traditional optical flow algorithms are based on single-scale image information in the processing of overlapping areas, which is difficult to fully capture the motion information of targets of different sizes, resulting in deviation of motion vectors, affecting the accuracy of geometric transformation matrix, and causing misalignment or deformation of the stitched image. At the same time, existing stitching methods lack sufficient processing of the seam and brightness difference in the overlapping area during image fusion, resulting in unnatural visual transition of the stitched image. Moreover, there is a lack of global optimization mechanism, making it difficult to use three-dimensional information of point clouds to correct optical flow estimation errors, resulting in inaccurate overall geometric relationship of the stitched image, especially in complex environments, making it difficult to guarantee the stitching accuracy and robustness.

[0005] Therefore, how to use the multi-scale characteristics of point clouds to accurately calculate the geometric transformation relationship of images combined with optical flow algorithms, improve the fusion quality of the overlapping area, and realize high-precision and robust line array camera image stitching is a problem to be solved. The present application aims to solve the above problems through multi-scale point cloud and image feature processing, optical flow fusion, geometric transformation matrix calculation and optimization, etc., to improve the stitching quality and reliability. SUMMARY

[0006] The present application aims to solve the problems existing in the prior art of linear array camera image stitching technology. In view of the defects of the traditional method, such as low matching rate in texture-deficient scene, insufficient precision of single-scale processing based on point cloud, deviation of motion vector calculation of optical flow algorithm, poor fusion effect of overlapping area, and lack of global optimization, through multi-scale point cloud and image feature cooperative processing, combined with optical flow fusion and accurate calculation of geometric transformation matrix, the overlapping area fusion is optimized, and high-precision and robust linear array camera image stitching is realized.

[0007] In view of the above defects or improvement needs of the prior art, as a first aspect of the present application, the present application provides a multi-scale linear array camera stitching method based on point cloud, comprising:

[0008] S1. Collect and preprocess the point cloud data and image data of the target area;

[0009] S2. Determine the overlapping area range between adjacent images through point cloud data and camera calibration information; extract spatial structure features in the point cloud data, and perform multi-scale hierarchical decomposition of the point cloud through a multi-scale segmentation method; correspondingly, perform multi-scale image feature extraction on the linear array camera image, and capture edge, texture and structure information at different scales;

[0010] S3. For any pixel in the overlapping area, use the optical flow method to solve the X direction gradient and the Y direction gradient, and calculate the motion vector between pixels; fuse the optical flow information obtained at each scale to construct a globally consistent optical flow vector field; and calculate the geometric transformation matrix of the entire overlapping area according to the fused optical flow vector;

[0011] S4. Perform fusion processing on the overlapping area after image transformation and alignment through the transformation matrix.

[0012] Further, the preprocessing process in S1 is specifically:

[0013] The collected point cloud and image data are preprocessed by an internal and external parameter correction method to realize the alignment of their spatial coordinates and time; and the continuous image data collected by the linear array camera are preprocessed including noise suppression, white balance adjustment and brightness normalization.

[0014] Further, the specific method for determining the overlapping area range between adjacent images in S2 is:

[0015] The point cloud is projected onto the plane of the adjacent image through the camera calibration parameters, specifically:

[0016] ,

[0017] ,

[0018] In the formula, is the coordinate of the three-dimensional point in the world coordinate system; is the coordinate of the three-dimensional point in the camera coordinate system; is a rotation matrix, is a three-dimensional translation vector; is the decomposed components; is an intrinsic matrix of the camera; is a focal length; is a principal point coordinate, which is the pixel coordinate of the intersection of the camera optical axis and the image plane; is the coordinate of the point in the image pixel coordinate system;

[0019] According to the projection point density, the overlapping area is determined, and finally the feature matching verification is completed to determine the range of the overlapping area of adjacent images.

[0020] Further, the specific method of determining the overlapping area according to the projection point density is:

[0021] The projection point density in the neighborhood of the pixel point on the image plane is defined as:

[0022] ,

[0023] wherein, is the pixel coordinate of the i-th projection point falling in the neighborhood ; is the neighborhood radius; is a Gaussian kernel parameter; and the high-density area is screened through a threshold

[0024] ,

[0025] wherein, is the image domain, is a candidate overlapping area.

[0026] Further, the specific method of the feature matching verification is:

[0027] The feature matching score of the points in the candidate area is calculated:

[0028] ,

[0029] wherein, is a region centered at the pixel , and the comprehensive score in the feature matching; is the pixel point of the current image to be calculated for the matching score; ​​​​​for the corresponding pixel point in the adjacent image ; for the pre-screened candidate matching region in the adjacent image is a spatial weight function is a feature descriptor is a feature similarity measure function

[0030] combining the point cloud projection density with the feature matching score to calculate the comprehensive confidence :

[0031] ,

[0032] wherein is a weight coefficient; and then determining the final overlapping region by a confidence threshold :

[0033] ,

[0034] wherein represents the final determined overlapping region.

[0035] Further, the multi-scale segmentation method in S2 is specifically:

[0036] Given the original point cloud , wherein is a three-dimensional space point

[0037] Set the multi-scale parameter , and construct a voxel grid :

[0038] ,

[0039] wherein is the feature value indexed by the two-dimensional discrete coordinates at the i-th scale, and the center point coordinate is ; represents an integer set

[0040] Map each point in the original point cloud to the corresponding voxel according to the formula , and record the voxel containing the point set ; calculate the barycenter of each point in the voxel to generate the sparse point cloud :

[0041] , ​

[0042] ,

[0043] wherein, represents the number of points in the voxel; sequentially generating point clouds of scales wherein , the larger the scale, the sparser the point cloud.

[0044] Further, the specific calculation method of the geometric transformation matrix in S3 is:

[0045] For the pixels in the overlapping area, the multi-scale optical flow vectors are fused using the equal weight average method:

[0046] ,

[0047] wherein, is the average optical flow vector of the final output; represents the total number of scales; represents the optical flow vector at the th scale;

[0048] In the overlapping area, evenly select feature points, record the original coordinates and the optical flow target coordinates wherein:

[0049] ,

[0050] ,

[0051] According to the affine transformation method, substitute feature point coordinates to construct an overdetermined linear equation set:

[0052] ,

[0053] wherein, is the linear transformation parameter; is the translation transformation parameter; the least square method is used to directly solve the parameters :

[0054] ,

[0055] wherein, , is the feature point coordinate matrix, is the target coordinate matrix; finally, the radiation transformation matrix is obtained:

[0056] ,

[0057] Recalculate the average error of the transformed feature points and the optical flow target coordinates When the threshold is exceeded; by manually removing the largest points of error, the affine matrix is recalculated.

[0058] Further, the average error is calculated by:

[0059] ,

[0060] wherein, is the transformed coordinate.

[0061] As a second aspect of the present application, a multi-scale point cloud-based linear array camera stitching system is also provided, comprising:

[0062] A data acquisition and preprocessing unit is configured to acquire and preprocess point cloud data and image data of a target region.

[0063] A multi-scale feature extraction unit is configured to determine the range of overlapping regions between adjacent images based on the point cloud data and camera calibration information, extract spatial structure features from the point cloud data, and perform multi-scale hierarchical decomposition of the point cloud data by using a multi-scale segmentation method. Correspondingly, multi-scale image feature extraction is performed on the linear array camera images to capture edge, texture, and structure information at different scales.

[0064] An optical flow vector calculation unit is configured to use an optical flow method to solve X-direction and Y-direction gradients for any pixel in the overlapping region and calculate the motion vector between pixels. The optical flow information obtained at different scales is fused to construct a globally consistent optical flow vector field. Based on the fused optical flow vector, a geometric transformation matrix of the entire overlapping region is calculated.

[0065] An image transformation and fusion unit is configured to perform fusion processing on the overlapping region after image transformation and alignment by using the transformation matrix.

[0066] As a third aspect of the present application, a computer readable storage medium having a computer program stored thereon is also provided. The computer program is executed by a processor to perform any step of the multi-scale point cloud-based linear array camera stitching method.

[0067] Overall, the above technical solutions conceived by the present application can achieve the following beneficial effects compared to the prior art:

[0068] 1. The point cloud-based multi-scale linear array camera stitching method of the present application determines the overlapping area range between adjacent images by using point cloud data and camera calibration information, simultaneously performs multi-scale hierarchical decomposition on the collected point cloud, extracts its spatial structure features, and makes the spatial distribution of the point cloud at different scales present; correspondingly, multi-scale feature extraction is carried out on the images obtained by the linear array camera, and edge, texture and structure information at each scale is captured. This process fully excavates the multi-scale features of the point cloud and the image, builds a cross-modal and multi-scale feature correlation system, lays a multi-dimensional feature foundation for the alignment and fusion of images in the subsequent stitching link, and guarantees the effective use and matching of features in different scale dimensions.

[0069] 2. The point cloud-based multi-scale linear array camera stitching method of the present application solves the X direction gradient and Y direction gradient by using the optical flow method for any pixel in the overlapping area, calculates the motion vector between pixels, and then fuses the optical flow information obtained at each scale to build a globally consistent optical flow vector field. Based on the fused optical flow vector, the geometric transformation matrix of the whole overlapping area is derived and calculated. The optical flow method captures the motion trend of the pixel, and the multi-scale optical flow fusion makes the vector field reflect the global motion law. The geometric transformation matrix converts the pixel motion relationship reflected by the optical flow into a mathematical model of image space transformation, provides a calculation basis for image alignment, realizes the connection from local pixel motion analysis to overall image space transformation modeling, solves the image misplacement problem caused by factors such as shooting angle and motion, and provides mathematical support for subsequent stitching.

[0070] 3. The point cloud-based multi-scale linear array camera stitching method of the present application completes the transformation and alignment of images by relying on the calculated geometric transformation matrix, and then performs fusion processing on the overlapping area. The geometric transformation matrix drives the image to adjust the spatial position, so that the adjacent images are aligned in the overlapping area, reducing the stitching misplacement caused by position deviation; based on the alignment result, the color and texture difference between images is smoothly transitioned in the overlapping area fusion link, reducing the stitching gap. This process takes multi-scale feature processing, optical flow field construction and matrix calculation as the premise, forms a complete closed loop of "feature extraction-transformation modeling-alignment fusion", effectively handles the problems such as image misplacement and obvious stitching marks in linear array camera image stitching, outputs a coherent and complete stitched image, realizes the goal of linear array camera multi-image stitching, and guarantees the unity of the stitched image in visual effect and spatial integrity. BRIEF DESCRIPTION OF DRAWINGS

[0071] Figure 1 The point cloud-based multi-scale linear array camera stitching method flowchart of the present application;

[0072] Figure 2 The stitched image schematic diagram of the present application;

[0073] Figure 3 System unit diagram of the embodiment of the present application. DETAILED DESCRIPTION

[0074] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0075] Embodiment 1

[0076] Please refer to Figure 1 The embodiment 1 provides a multi-scale line array camera splicing method based on point cloud, comprising:

[0077] S1. Complete the acquisition and preprocessing of point cloud data and image data of the target area;

[0078] S2. Determine the overlapping area range between adjacent images through point cloud data and camera calibration information; extract spatial structure features in the point cloud data, and perform multi-scale hierarchical decomposition of the point cloud through a multi-scale segmentation method; correspondingly, perform multi-scale image feature extraction on the line array camera image, and capture edge, texture and structure information at different scales;

[0079] S3. For any pixel in the overlapping area, use the optical flow method to solve the X direction gradient and the Y direction gradient, and calculate the motion vector between the pixels; and fuse the optical flow information obtained at each scale to construct a globally consistent optical flow vector field; according to the fused optical flow vector, calculate the geometric transformation matrix of the whole overlapping area;

[0080] S4. After image transformation and alignment through the transformation matrix, perform fusion processing on the overlapping area.

[0081] The embodiment 1 further expands the above steps.

[0082] (1) Data acquisition and preprocessing

[0083] The embodiment 1 acquires point cloud data and image data of the target area through synchronous acquisition and preprocessing. Laser scanning or structured light sensor is used to acquire point cloud data, and records the three-dimensional space coordinates of the object surface; the line array camera acquires image data in a continuous shooting mode, and obtains texture and color information of the target area.

[0084] After data acquisition, the internal and external parameter correction method is used for pretreatment. For image data, noise is removed by filtering method, color deviation is adjusted by white balance, and brightness normalization processing is performed to unify the brightness difference between images, providing stable texture information for optical flow calculation. For point cloud data, the spatial mapping relationship between point cloud and image is established by accurately calibrating the camera internal parameters (focal length, principal point position, etc.) and external parameters (camera spatial position and attitude), and a three-dimensional geometric model is constructed, providing a spatial reference for determining the image overlap area and image correction.

[0085] Through the above pretreatment, the point cloud data and image data are aligned in space coordinates and time, the data deviation is eliminated, and the data quality is improved, providing a reliable data basis for subsequent operations such as determining the overlap area range, extracting multi-scale features, calculating optical flow and geometric transformation matrix, etc.

[0086] (2) Multi-scale feature extraction

[0087] In this embodiment 1, the point cloud data and camera calibration information are used to determine the overlap area range of adjacent images, and multi-scale feature extraction is performed. The point cloud is projected to the adjacent image plane through the camera calibration parameters, the candidate overlap area is preliminarily screened according to the projection point density, and the comprehensive confidence is calculated through feature matching verification, and finally the overlap area range is determined, which limits the area for subsequent optical flow calculation.

[0088] In a preferred embodiment, the specific method for determining the overlap area range between adjacent images is as follows:

[0089] The point cloud is projected to the adjacent image plane through the camera calibration parameters, specifically as follows:

[0090] ,

[0091] ,

[0092] In the formula, is the coordinate of the three-dimensional point in the world coordinate system; is the coordinate of the three-dimensional point in the camera coordinate system; is the rotation matrix, is the three-dimensional translation vector; is the expanded component; is the camera intrinsic parameter matrix; is the focal length; is the principal point coordinate, which is the pixel coordinate of the intersection of the camera optical axis and the image plane; is the coordinate of the point in the image pixel coordinate system;

[0093] Then the overlap area is determined according to the projection point density, and finally the feature matching verification is completed to determine the overlap area range of adjacent images.

[0094] In the preferred embodiment, the specific method of determining the overlap region according to the density of the projected points is as follows:

[0095] The density of the projected points in the neighborhood of the pixel point on the image plane is defined as:

[0096]

[0097] wherein is the pixel coordinate of the th projected point in the neighborhood ; is the neighborhood radius; is the Gaussian kernel parameter; and the high-density region is screened by a threshold

[0098]

[0099] wherein is the image domain, and is the candidate overlap region.

[0100] In the preferred embodiment, the specific method of feature matching verification is as follows:

[0101] The feature matching score of the points in the candidate region is calculated as:

[0102]

[0103] wherein is the region centered on the pixel , and is the comprehensive score in feature matching; is the pixel point in the current image to be calculated for the matching score; is the corresponding pixel point in the adjacent image; is the pre-screened candidate matching region in the adjacent image; is the spatial weight function; is the feature descriptor, and is the feature similarity measure function;

[0104] The comprehensive confidence is calculated by combining the density of the projected points and the feature matching score :

[0105]

[0106] wherein is the weight coefficient; and the confidence threshold is used for screening.​​​​​​​​​​​ determining the final overlapping region:

[0107] ,

[0108] wherein, represents the final determined overlapping region.

[0109] In the aspect of point cloud multi-scale segmentation, given an original three-dimensional point cloud, a multi-scale parameter is set to construct a voxel grid, each point is mapped to a corresponding voxel according to the scale parameter, the center of gravity of the points in each voxel is calculated, and sparse point clouds of different scales are generated. The larger the scale is, the sparser the point cloud is, realizing multi-scale hierarchical decomposition of the point cloud and extracting spatial structure features.

[0110] In a preferred embodiment, the multi-scale segmentation method is specifically:

[0111] Given an original point cloud wherein, is a three-dimensional space point;

[0112] a multi-scale parameter is set , a voxel grid is constructed :

[0113] ,

[0114] wherein, is the feature value indexed by the two-dimensional discrete coordinate at the i-th scale, and the center point coordinate is ; represents an integer set; Each point

[0115] in the original point cloud is mapped to a corresponding voxel according to the formula , and the set of points contained in the voxel is recorded as ; the center of gravity of the points in each voxel is calculated, and a sparse point cloud is generated:

[0116] ,

[0117] ,

[0118] wherein, represents the number of points in the voxel; point clouds of scales are generated in turn , wherein , the larger the scale is, the sparser the point cloud is.

[0119] ​For images from a line scan camera, a multi-scale pyramid is constructed. Starting from the original image, the image is smoothed with a Gaussian kernel and then downsampled at alternating rows and columns to generate multi-layer images. Extreme points are found by using the difference of Gaussians. Regions are divided with key points as the center, and gradient histograms are calculated to generate descriptors. At the same time, the FAST method is used to filter key points. The grayscale of pixels in the neighborhood of key points is compared to generate binary descriptors, thereby realizing multi-scale image feature extraction and capturing edge, texture and structural information at different scales.

[0120] Furthermore, for each scale ( ), to point cloud Corresponding image pyramid levels Establish connections; utilize camera intrinsic parameters and external references , will point Projected onto the image plane:

[0121] ,

[0122] in, These are pixel coordinates; at the image layer In the middle, extraction point Nearby image features It is associated with point cloud features (such as density and height difference).

[0123] (3) Calculation of optical flow vector

[0124] This embodiment 1 achieves pixel motion analysis and image transformation modeling in overlapping regions through multi-scale optical flow estimation and geometric transformation matrix calculation. In the multi-scale optical flow estimation stage, for pixels in the overlapping region, the X and Y direction gradients are solved using optical flow methods at each level of the image pyramid to calculate the motion vectors between pixels. Specifically, a coarse optical flow field is calculated starting from the coarsest level, upsampled to the next detail level as an initial value for optimization, and refined layer by layer until a pixel-level optical flow field at full resolution is obtained, reflecting the correspondence between any pixels in adjacent images. Then, optical flow information from each scale is fused to construct a globally consistent optical flow vector field, used to represent the offset of pixels in the overlapping region.

[0125] During the calculation of the geometric transformation matrix, the multi-scale optical flow vectors of pixels in the overlapping region are fused using an equal-weighted averaging method to obtain the average optical flow vector. N feature points are uniformly selected in the overlapping region, and their original coordinates and the target coordinates calculated based on the average optical flow vector are recorded. According to the affine transformation principle, these feature point coordinates are substituted into a system of overdetermined linear equations, and the affine transformation parameters are solved using the least squares method to obtain the affine transformation matrix. The average error between the transformed feature points and the optical flow target coordinates is calculated. If the error exceeds a threshold, the K points with the largest errors are manually removed, and the affine matrix is ​​recalculated until the requirements are met.

[0126] In the preferred embodiment, the specific calculation method of the geometric transformation matrix is as follows:

[0127] For the pixels in the overlapping area, the multi-scale optical flow vectors are fused by using the equal-weight average method:

[0128] ,

[0129] In the formula, is the average optical flow vector of the final output; represents the total number of scales; represents the optical flow vector at the i-th scale;

[0130] In the overlapping area, a plurality of feature points are uniformly selected, and the original coordinates and the optical flow target coordinates are recorded. Among them:

[0131] ,

[0132] ,

[0133] According to the affine transformation method, the coordinates of the feature points are substituted to construct an overdetermined linear equation set:

[0134] ,

[0135] In the formula, is the linear transformation parameter; is the translation transformation parameter; the least square method is used to directly solve the parameters :

[0136] ,

[0137] Among them, , is the feature point coordinate matrix, is the target coordinate matrix; and the final affine transformation matrix is obtained:

[0138] ,

[0139] Then, the average error between the transformed feature points and the optical flow target coordinates is calculated When the average error exceeds the threshold value; by manually removing the points with the largest error, the affine matrix is recalculated.

[0140] In the preferred embodiment, the calculation method of the average error is as follows:​​​​​

[0141] ,

[0142] wherein, is the transformed coordinate.

[0143] (4) Image transformation fusion

[0144] Referring to Figure 2 , after the completion of the geometric transformation matrix calculation, the embodiment 1 enters the image transformation and alignment and fusion processing link. The target image is subjected to geometric transformation by using the calculated transformation matrix, the pixel coordinates of the target image are mapped to the positions corresponding to the overlapping area of the reference image through matrix operation, the spatial positions of adjacent images in the overlapping area are aligned, and the pixel positions of the same scene content in the two images are accurately corresponded.

[0145] For the overlapping area, image fusion methods such as weighted average and Poisson fusion are used for processing. The weighted average assigns weights according to the distance of the pixel to the boundary of the overlapping area, so that the edge pixels are smoothly transitioned; the Poisson fusion realizes natural connection of texture by calculating the gradient field, effectively eliminates the splicing joint and brightness difference, makes the image color and texture of the overlapping area transition naturally, and obtains a visually smooth and coherent splicing effect.

[0146] If there is three-dimensional point cloud data, the preliminary splicing result is combined with the three-dimensional model constructed by the point cloud, and the error is further corrected through global geometric optimization. The point cloud data provides the three-dimensional spatial structure information of the scene, associates the pixel points in the spliced image with the point cloud coordinates in the three-dimensional model, adjusts the image transformation parameters through the optimization method, corrects the slight misplacement caused by the optical flow estimation error, ensures that the overall geometric relationship of the spliced image is consistent with the actual scene, and improves the spatial accuracy of splicing.

[0147] The multi-scale features after fusion are used to automatically generate a global splicing model. The multi-scale features contain image details and point cloud structure information at different resolutions, and the model models the geometric relationship of the entire spliced image based on these features, identifies and locates local splicing error areas, compensates and optimizes the errors through parameter adjustment, and further improves the overall accuracy of the spliced image.

[0148] In addition, an online feedback and iterative updating strategy is designed. In actual application, the system continuously collects new point cloud and image data, compares and analyzes with the spliced result, detects the change of splicing error. According to the feedback information, the parameters and model of the splicing method are iteratively updated, which adapts to environmental changes, ensures that in complex scenes such as light mutation and object motion, the splicing accuracy and robustness can still be maintained, and makes the splicing system have the ability of continuous optimization.

[0149] Embodiment 2

[0150] Please refer to Figure 3 The embodiment 2 provides a point cloud-based multi-scale linear array camera splicing system, comprising the following steps:

[0151] A data acquisition and preprocessing unit is configured to acquire and preprocess point cloud data and image data of a target region.

[0152] A multi-scale feature extraction unit is configured to determine an overlapping area range between adjacent images based on the point cloud data and camera calibration information, extract spatial structure features in the point cloud data, and perform multi-scale hierarchical decomposition on the point cloud data by using a multi-scale segmentation method; correspondingly, multi-scale image feature extraction is performed on the linear array camera images to capture edge, texture and structure information at different scales.

[0153] An optical flow vector calculation unit is configured to solve X-direction gradient and Y-direction gradient by using an optical flow method for any pixel in the overlapping area, and calculate a motion vector between the pixels; the optical flow information obtained at different scales is fused to construct a globally consistent optical flow vector field; and a geometric transformation matrix of the entire overlapping area is calculated based on the fused optical flow vector.

[0154] An image transformation and fusion unit is configured to perform fusion processing on the overlapping area after image transformation and alignment by using the transformation matrix.

[0155] Embodiment 3

[0156] The embodiment 3 further provides a computer readable storage medium having a computer program stored thereon, and the computer program, when executed by a processor, can implement any step of the point cloud-based multi-scale linear array camera splicing method.

[0157] The computer readable storage medium can include a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0158] The computer readable storage medium provided in the present application is described in the above method embodiments, and will not be repeated here.

[0159] Those skilled in the art can easily understand that the above description is only a preferred embodiment of the present application, and is not intended to limit the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-scale linear array camera stitching method based on point clouds, characterized in that, include: S1. Complete the acquisition and preprocessing of point cloud data and image data of the target area; S2. Determine the overlapping area between adjacent images using point cloud data and camera calibration information; extract spatial structure features from the point cloud data, and decompose the point cloud into multi-scale hierarchical components using a multi-scale segmentation method; correspondingly, extract multi-scale image features from the line scan camera images to capture edge, texture, and structural information at different scales. S3. For any pixel in the overlapping region, use the optical flow method to solve the gradient in the X and Y directions, and calculate the motion vector between pixels; then fuse the optical flow information obtained at each scale to construct a globally consistent optical flow vector field; and calculate the overall geometric transformation matrix of the overlapping region based on the fused optical flow vector. S4. After image transformation and alignment using a transformation matrix, the overlapping areas are then fused. The multi-scale segmentation method is specifically as follows: Given the original point cloud ,in A point in three-dimensional space; Setting multi-scale parameters Constructing a voxel mesh : ， in, For the first At each scale, using two-dimensional discrete coordinates The eigenvalues ​​of the index, whose center point coordinates are... ; Represents the set of integers; Original point cloud Each point in According to the formula Map to the corresponding voxel, and record the voxel. The set of points included is ; Calculate the centroid for each point within each voxel. Generate a sparsed point cloud : ， ， In the formula, Indicates the number of points within a voxel; generated sequentially. Point clouds at various scales ,in The larger the scale, the sparser the point cloud.

2. The multi-scale linear array camera stitching method based on point clouds according to claim 1, characterized in that, The preprocessing process in S1 is specifically as follows: The acquired point cloud and image data are preprocessed using intrinsic and extrinsic parameter correction methods to achieve spatial and temporal alignment between the two; and the continuous image data acquired by the line scan camera is preprocessed including noise suppression, white balance adjustment and brightness normalization.

3. The multi-scale linear array camera stitching method based on point clouds according to claim 1, characterized in that, The specific method for determining the overlapping region range between adjacent images in S2 is as follows: The point cloud is projected onto an adjacent image plane using camera calibration parameters, specifically as follows: ， ， In the formula, These are the coordinates of a 3D point in the world coordinate system. These are the coordinates of a 3D point in the camera coordinate system. For rotation matrix, It is a three-dimensional translation vector; for The expanded components; This is the camera intrinsic parameter matrix; Focal length; Principal point coordinates are the pixel coordinates of the intersection of the camera's optical axis and the image plane. The coordinates of the point in the image pixel coordinate system; Then, the overlapping area is determined based on the density of the projection points, and finally, the range of the overlapping area between adjacent images can be determined by completing the feature matching verification.

4. The multi-scale linear array camera stitching method based on point clouds according to claim 3, characterized in that, The specific method for determining the overlapping region based on the projection point density is as follows: Define pixels on the image plane neighborhood Projection point density within for: ， in, To fall in the neighboring area The first in The pixel coordinates of each projection point; The neighborhood radius; The parameters are Gaussian kernel parameters; then a threshold is applied. Filter high-density areas: ， in, For the image domain, These are candidate overlapping regions.

5. The multi-scale linear array camera stitching method based on point clouds according to claim 3, characterized in that, The specific method for feature matching verification is as follows: Calculate candidate regions Interior point feature matching score: ， in, In pixels The overall score for feature matching in the central region; The pixels in the current image for which matching scores are to be calculated; For adjacent images with The corresponding pixels; These are pre-selected candidate matching regions in adjacent images; This is the spatial weighting function; For feature descriptors, This is a feature similarity measurement function; Combined point cloud projection density Feature matching score Calculate the overall confidence level : ， In the formula, These are the weighting coefficients; then, they are passed through a confidence threshold. Determine the final overlapping area: ， In the formula, This indicates the final determined overlapping area.

6. The multi-scale linear array camera stitching method based on point clouds according to claim 1, characterized in that, The specific calculation method for the geometric transformation matrix in S3 is as follows: For pixels in overlapping regions, fuse multi-scale optical flow vectors. The equal-weighted average method is used: ， In the formula, This is the final output average optical flow vector; Indicates the total number of scales; Indicates the first Optical flow vectors at various scales; Select uniformly in the overlapping region Each feature point records its original coordinates. Coordinates of optical flow target ,in: ， ， According to the affine transformation method, Substituting the coordinates of the feature points, we construct an overdetermined system of linear equations: ， In the formula, These are the parameters for linear transformation; The parameters are for translation transformation; the parameters are solved directly using the least squares method. : ， in, , The feature point coordinate matrix The target coordinate matrix is ​​obtained; the final radial transformation matrix is ​​obtained. : ， Then calculate the average error between the transformed feature points and the optical flow target coordinates. ,when When the threshold is exceeded, manually remove the errors with the largest errors. For each point, recalculate the affine matrix.

7. The multi-scale linear array camera stitching method based on point clouds according to claim 6, characterized in that, The average error The calculation method is as follows: ， in, These are the transformed coordinates.

8. A multi-scale linear array camera stitching system based on point clouds, characterized in that, include: The data acquisition and preprocessing unit is used to acquire and preprocess point cloud data and image data of the target area. The multi-scale feature extraction unit is used to determine the overlapping area between adjacent images through point cloud data and camera calibration information; and to extract spatial structure features from the point cloud data, and to decompose the point cloud into multi-scale hierarchical components through a multi-scale segmentation method; correspondingly, multi-scale image feature extraction is performed on the linear array camera image to capture edge, texture and structural information at different scales. The optical flow vector calculation unit is used to solve the X-direction gradient and Y-direction gradient for any pixel in the overlapping region using the optical flow method, and to calculate the motion vector between pixels; it also fuses the optical flow information obtained at each scale to construct a globally consistent optical flow vector field; and calculates the overall geometric transformation matrix of the overlapping region based on the fused optical flow vector. The image transformation and fusion unit is used to perform image transformation and alignment using a transformation matrix, and then fuse the overlapping areas. The multi-scale segmentation method is specifically as follows: Given the original point cloud ,in A point in three-dimensional space; Setting multi-scale parameters Constructing a voxel mesh : ， in, For the first At each scale, using two-dimensional discrete coordinates The feature value of the index, whose center point coordinates are ; Represents the set of integers; Original point cloud Each point in According to the formula Map to the corresponding voxel, and record the voxel. The set of points included is Calculate the centroid for each point within each voxel. Generate a sparsed point cloud : ， ， In the formula, Indicates the number of points within a voxel; generated sequentially. Point clouds at various scales ,in The larger the scale, the sparser the point cloud.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is executed by a processor according to any one of claims 1-7, which describes a multi-scale linear array camera stitching method based on point clouds.

Images