Real-time three-dimensional fusion reconstruction method for ship comprehensive test scene

Abstract

The invention discloses a real-time three-dimensional fusion reconstruction method for a ship comprehensive test scene, and the method comprises the steps: carrying out the preprocessing of a target object image data flow, converting an image coordinate system into a camera coordinate system, obtaining a sparse point cloud model, and constructing a voxel three-dimensional grid; the method comprises the following steps: constructing a scale space by using a Gaussian difference pyramid to detect key points of an image under different scales, calculating a gradient direction histogram around the key points and taking the gradient direction histogram as a descriptor, matching the descriptor by using an Euclidean distance, and estimating relative pose transformation between the images to complete a local matching process. Projecting and mapping the local frame block to a reference frame to form a key frame, and entering a global matching process; and extracting a surface grid of the three-dimensional model from the dense body grid by adopting a Marching Cubes algorithm, traversing each cube unit in the surface grid, and obtaining distance information from a vertex to a nearest surface based on the cube units, thereby generating a final three-dimensional model of the test target object.

Description

Technical Field

[0001] The present invention relates to the technical field of test scene reconstruction, and in particular to a real-time three-dimensional fusion reconstruction method for a ship comprehensive test scene. Background Art

[0002] With the rapid development of intelligent ship technology, real-time reconstruction of ship comprehensive test scenarios has become an important research direction for navigation safety and intelligent navigation. During sea navigation, ships need to interact and avoid collisions with other ships and the environment to ensure the safety and reliability of navigation. Therefore, accurately reconstructing and analyzing ship comprehensive test scenarios is crucial for evaluating the collision avoidance strategies of intelligent ships and optimizing navigation paths. Most of the existing ship comprehensive test scenario reconstruction methods are based on offline data analysis or virtual simulation technology. However, these methods have some limitations, such as the inability to obtain real-world data in real time, difficulty in simulating real ship motion, and the inability to provide accurate scenario analysis. Summary of the invention

[0003] In view of the problems existing in the prior art, the present invention discloses a real-time three-dimensional fusion reconstruction method for a ship comprehensive test scene, which specifically includes the following steps:

[0004] Read the internal parameter information of the RGBD camera and the monocular camera, collect the target image data stream of the ship test scene based on the RGBD camera and the monocular camera, pre-process the target image data stream to convert the image coordinate system to the camera coordinate system and obtain a sparse point cloud model to construct a voxel three-dimensional grid.

[0005] Use Gaussian difference pyramid to construct scale space to detect key points of images at different scales, calculate the gradient direction histogram around the key points and use it as a descriptor, use Euclidean distance to match the descriptor, estimate the relative pose transformation between images to complete the local matching process, project the local frame block to the reference frame to form a key frame, and enter the global matching process: including using SIFT algorithm to detect common feature points between key frames and estimate camera pose, iteratively optimize the camera pose and point cloud position based on Gauss-Newton algorithm and minimize the reprojection error, update the pose information of the key frame according to the optimized camera pose and point cloud position, and fuse the voxel values ​​in the TSDF function to form a dense volumetric grid;

[0006] The Marching Cubes algorithm is used to extract the surface mesh of the three-dimensional model from the dense volume mesh. Each cube unit in the surface mesh is traversed, and the distance information from the vertex to the nearest surface is obtained based on the cube unit, thereby generating the final three-dimensional model of the test target.

[0007] Furthermore, local matching of the target object image data stream includes: extracting SIFT feature points from the depth image based on the SIFT algorithm, finding key points in the scale space in the image, determining the position and scale of the SIFT feature points by detecting stable feature points in the image, constructing the scale space using a Gaussian difference pyramid, detecting key points in each scale space, determining key points by finding local extreme points, calculating feature descriptors for the neighborhood area of ​​each key point, generating a gradient direction histogram near the key point to describe the image information around the key point, generating a descriptor vector using the image gradient direction and intensity information near the key point, matching the feature descriptor vectors in the two images, finding corresponding feature point pairs in different images, measuring the similarity between the two feature vectors using the Euclidean distance, selecting the best matching point pair using the nearest neighbor matching method, estimating the relative pose transformation between the two images using the matched feature point pairs, iteratively selecting a random sample set and calculating the pose transformation model, evaluating the degree of conformity of all matching point pairs with the sparse point cloud model, and selecting the model that minimizes the error between the point clouds as the optimal pose transformation model.

[0008] Furthermore, the following method is used to form a dense volumetric grid: feature point matching and pose estimation are performed on the image keyframe based on the SIFT algorithm, and the camera pose and point cloud position are iteratively optimized based on the Gauss-Newton algorithm. In each iteration, the Jacobian matrix J and the residual vector r are calculated, and the Hessian matrix is ​​calculated. The reprojection error function is defined as:

[0009] E(p)=∑w*‖p′-p‖ 2

[0010] Where p′ is the predicted feature point coordinate, p is the observed feature point coordinate, and w is the weight. The parameters of the pose and point cloud position are iteratively updated in the form of minimizing the reprojection error until the convergence condition is reached, thereby reducing the difference between the projection position and the actual observed image coordinates. The pose information of the key frame is updated according to the optimized pose and point cloud position, and the voxel values ​​in the TSDF function are fused to form a dense volumetric grid. When the TSDF value and weight reach a certain threshold, the voxel is considered to have been stabilized and fused into the global model.

[0011] Furthermore, when extracting the surface mesh of the three-dimensional model from the dense volumetric mesh: first define an empty triangular mesh data structure to store the generated triangular facets, traverse each voxel unit of the voxel three-dimensional grid, determine the triangular facets that need to be generated on the voxel boundary based on the combination of internal and external labels of the voxel, use the table lookup method for acceleration and optimization, and finally obtain the surface mesh.

[0012] Furthermore, when reading the internal parameters of the RGBD camera and the monocular camera, the camera lens distortion correction is performed, and the distortion parameters are used for dedistortion processing; when preprocessing the target image data stream, it includes: setting the filter window size, calculating the spatial distance and grayscale difference between each pixel in each filter window and the central pixel, obtaining the spatial weight and grayscale weight, obtaining the comprehensive weight based on the product of the spatial weight and the grayscale weight, performing weighted averaging processing on the pixels in the filter window according to the comprehensive weight to obtain the filtered pixel value, sliding the filter window on the image to cover the entire image, and then performing an average pooling operation on the image information after bilateral filtering.

[0013] The present invention provides a real-time three-dimensional fusion reconstruction method for a ship comprehensive test scene. The method adopts the BundleFusion algorithm to achieve real-time three-dimensional reconstruction, and its parallel computing capability can accelerate the processing and modeling process of point cloud data. Compared with some traditional methods, Bundle Fusion can generate and update the three-dimensional model of the ship comprehensive test scene more quickly, thereby providing more timely information. Bundle Fusion utilizes global optimization technology to help improve the accuracy of three-dimensional reconstruction. By considering the consistency of the entire scene, the method can accurately restore the environment around the ship, including the water surface, other ships and surrounding objects.

[0014] Ship comprehensive test scenarios usually contain a large number of dynamic changes in the scene, such as waves, ship movement, etc. BundleFusion has a certain adaptability to these dynamic changes and can better handle moving objects, making the scene around the ship more realistic and complete. Because the Bundle Fusion algorithm uses a voxel-based representation, this innovative data expression method shows excellent scalability when dealing with large-scale and complex ship test scenarios. The voxel representation not only improves the speed and efficiency of data processing, but also ensures the accuracy and stability of scene reconstruction. This feature is particularly critical when dealing with complex simulated marine environments. Ship comprehensive test scenarios often involve vast water areas and multiple ships participating in the experiment. The voxel representation of the Bundle Fusion algorithm can quickly and accurately capture every detail. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1Flow chart of the method of the present invention

[0017] Figure 2 A schematic diagram of a truncation-based signed distance function according to an embodiment of the present invention;

[0018] Figure 3 This is a schematic diagram of local matching according to an embodiment of the present invention;

[0019] Figure 4 A schematic diagram of global optimization according to an embodiment of the present invention;

[0020] Figure 5 A schematic diagram of the Marching Cubes algorithm described in an embodiment of the present invention;

[0021] Figure 6 Schematic diagram of the hierarchical clustering method described in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the technical solutions and advantages of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention:

[0023] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] like Figure 1 A real-time 3D fusion reconstruction method for a ship comprehensive test scene is shown, which specifically includes the following steps:

[0026] like Figure 1-6As shown, the present invention provides a real-time reconstruction method for a ship comprehensive test scenario based on Bundle Fusion, comprising the following steps:

[0027] Based on the RGBD camera and the monocular camera, the image information data stream of the target object in the ship comprehensive test scene is collected, and the camera is calibrated based on the calibration board to obtain the camera intrinsic parameters and distortion. The camera intrinsic parameters are expressed by the following formula:

[0028]

[0029] u and v are pixel coordinates, indicating the position of the point on the camera imaging plane; X, Y, and Z are the positions of the point in the world coordinate system; K is the camera intrinsic parameter matrix, indicating the internal parameters of the camera. The general form of the camera intrinsic parameter matrix K is as follows:

[0030]

[0031] f x and f y is the focal length of the camera, which represents the ratio of pixels to actual distance; c x and c y is the principal point coordinate, which indicates the position of the principal point on the camera imaging plane.

[0032] In order to eliminate the influence of distortion on image quality, the camera lens distortion correction is performed by using the distortion parameters for dedistortion. The formula is as follows:

[0033]

[0034] In the above formula, (u corrected ,v corrected ) is the coordinate after dedistortion, r is the radial distance, and the calculation formula is: k 1 , k 2 , k 3 is the radial distortion coefficient, which is used to describe the degree of radial distortion, p 1 , p 2 is the degree of tangential distortion coefficient. Based on the camera calibration, the distortion parameters of the camera are obtained, the distortion correction formula is applied to the coordinates of each pixel point, and the distorted coordinates are mapped back to the original image coordinate system.

[0035] After obtaining the corrected image information data stream, preprocessing is performed. Through bilateral filtering, edge information is maintained while image noise is reduced. The filter window is set to 5*5. For each pixel in the filter window, the spatial distance between it and the center pixel is calculated. The spatial weight takes into account the spatial relationship between pixels. The farther the pixel is from the center pixel, the lower the weight. For each pixel in the filter window, the grayscale difference between it and the center pixel is calculated. The smaller the grayscale difference, the higher the pixel weight. Multiply the spatial weight and the grayscale weight to obtain the comprehensive weight. The pixels in the filter window are weighted averaged according to the comprehensive weight to obtain the filtered pixel value. Slide the filter window on the image to cover the entire image. Perform a filtering operation on each window and store the filtering result in the output image.

[0036] The image information after bilateral filtering is averaged and pooled. The image pixels are reduced to 640*480, and the pooling window is set to 3*3. The window will slide on the image and the pixels in each window will be pooled. For each pooling window, the average value of the pixels in the window is calculated. The values ​​of all pixels in the window are added and then divided by the total number of pixels in the window to get the average value. This average value will be used as the output value of the pooling window. The pooling window is slid on the image to cover the entire image. The stride is fixed to 2 and slides, which means sliding 2 pixels each time. Pooling is performed on each window and the pooling result is stored in the output image. The pooling operation will reduce the size of the original image to 640*480.

[0037] After image preprocessing, for pixels in the depth map that represent valid depth information, use the following formula to convert them from the image coordinate system to the camera coordinate system:

[0038]

[0039] Among them, (u,v) is the pixel coordinate in the image coordinate system, (X,Y,Z) is the coordinate in the camera coordinate system, and Z is the distance from the pixel to the camera obtained from the depth image. Combine the converted three-dimensional coordinates (X,Y,Z) with the corresponding RGB color values ​​to form a sparse point cloud model. Each point cloud point contains position information (X,Y,Z) and color information (R,G,B). Construct a voxel three-dimensional grid and assign an initial distance value and weight to each voxel unit. Calculate the signed distance d from the point cloud p to the nearest surface. Update the TSDF value and weight. The update formula of TSDF is:

[0040]

[0041] The formula for weight update is:

[0042] weight_new=weight_old+1

[0043] The TSDF value range is limited to [-1, 1] by the truncation function.

[0044] Perform local matching on the preprocessed image, such as Figure 2 As shown, SIFT feature points are extracted from the depth image based on the SIFT algorithm and matched with the previous frames in the local frame block. If the current frame does not match, a new local frame block is opened and 10 frames are supplemented by blank frames. Based on the SIFT algorithm, the first step is feature point detection, which searches for key points in the scale space in the image. The position and scale of the SIFT feature are determined by detecting stable feature points in the image. The scale space is constructed using a Gaussian difference pyramid. The construction formula of the Gaussian difference pyramid is as follows:

[0045] G(x,y,σ)=I(x,y)*G(x,y,σ)

[0046] Where G(x,y,σ) is a Gaussian pyramid, I(x,y) is the original image, * indicates a convolution operation, and G(x,y,σ) is a Gaussian kernel. Key point detection is performed on each scale space, and key points are determined by finding local extreme points. The feature descriptor is calculated for the neighborhood area of ​​each key point, and a gradient direction histogram near the key point is generated to describe the image information around the key point. The descriptor vector is generated using the image gradient direction and intensity information near the key point. The feature descriptor is calculated as follows:

[0047] H(x,y,θ)=σw(x,y,σ)*d(x,y,θ)

[0048] Among them, H(x,y,θ) is the gradient direction histogram, w(x,y,σ) is the Gaussian weighting function, and d(x,y,θ) is the weighting of the gradient amplitude. The descriptor vector is scale-invariant and rotation-invariant, and can resist illumination changes and perspective changes to a certain extent. Match the feature descriptors in the two images to find the corresponding feature point pairs in different images. Use Euclidean distance to measure the similarity between two feature vectors. Use the nearest neighbor matching method to select the best matching point pairs. Estimate the relative pose transformation between the two images based on the matched feature point pairs based on the RANSAC algorithm. RANSAC iteratively selects a random sample set and calculates the pose transformation model, then evaluates the degree of conformity of all matching point pairs with the sparse point cloud model, and finally selects the model that minimizes the error between the point clouds as the optimal pose transformation model.

[0049] The same feature points of the local frame block are projected and mapped into the reference frame to form a key frame. The common feature points between the current frame and the previous frames in the local frame block are found and globally matched to reduce the difference between the projected position and the actual observed image coordinates, such as Figure 3As shown. Based on the SIFT algorithm, the key frames are matched with feature points and the pose is estimated. Based on the Gauss-Newton algorithm, the camera pose and point cloud position are iteratively optimized. In each iteration, the Jacobian matrix J and the residual vector r are calculated. The Hessian matrix, J T * J and g = J T *r, update parameter increment △x=(H+λ) -1 *g, where λ is the damping coefficient. Update the parameters x=x+△x. Step by step optimize them to minimize the reprojection error. The reprojection error function is defined as:

[0050] E(p)=∑w*‖p′-p‖ 2

[0051] where p ′ is the predicted feature point coordinate, p is the observed feature point coordinate, and w is the weight. The form of minimizing the reprojection error is adopted. The parameters of the pose and point cloud position are iteratively updated until the convergence condition is reached, effectively reducing the difference between the projection position and the actual observed image coordinates. According to the optimized pose and point cloud position, the pose information of the key frame is updated, and the voxel values ​​in the TSDF function are updated and fused to form a dense voxel grid. When the TSDF value and weight reach a certain threshold, the voxel is considered to be stable and integrated into the global model.

[0052] Voxel surface extraction,For dense TSDF data, voxels are marked as internal or external by threshold judgment. For voxels located between the internal and external boundaries, the Marching Cubes algorithm is used to generate a triangular mesh to represent the voxel surface. Figure 5 As shown. Define an empty triangular mesh data structure to store the generated triangular facets. Traverse each voxel unit of the voxel data. According to the combination of the voxel internal and external marks, determine the triangular facets that need to be generated on the voxel boundary, and use the table lookup method for acceleration and optimization. For each triangular facet that needs to be generated, interpolate the position of the vertex on the boundary according to the voxel internal and external marks. Assume that there are two vertices v on the voxel boundary. 1 and v 2 , their inner and outer labels are f 1 and f 2 . Calculate the interpolation ratio θ is the threshold value. The result of interpolation calculation is the internal and external marks on the boundary point f = t × f 2 +(1-t)×f 1 . Determination of vertex positions. Assume that there are two vertices on the voxel boundary, and their positions are p 1 and p 2 The result of the interpolation calculation is the vertex position p = t × p 2 +(1-t)×p 1The vertex index of the triangle is determined according to the vertex position and the inside and outside marks of the voxel. The generated triangle is added to the triangle mesh data structure.

[0053] Based on the above steps, the target object model of the test scene is obtained and the model is post-processed. The resolution of the depth map is 640*480, and equal division is set. The length is divided into 640 / 10=64, and the width is divided into 480 / 10=48, totaling 100 parts. Each block contains n=64*48 depth nodes. Then, in order to facilitate subsequent processing, some invalid depth nodes are removed. The depth information contained in the node is incomplete, the depth information of the parent node is incomplete, the corresponding edge is not in this plane, and the nodes with too high or too low out-degree and in-degree are all removed, thereby generating a new node set. Based on the hierarchical clustering algorithm, the foreground area is identified and segmented, such as Figure 6 As shown in the figure, the plane model is fitted by the RANSAC algorithm. The general equation of the plane model is: Ax+By+Cz=0, where A, B, C and D are the parameters of the plane model. The distance from each point to the estimated plane model is calculated by the Euclidean distance. A threshold is set, and the points whose distance from the estimated model is within the threshold range are defined as inliers for further optimization of the model. The quality of the current model is evaluated according to the number of inliers. Through multiple iterations of sampling, estimation and evaluation, the optimal plane model is found, and the optimal plane model parameter set PI is output. The similarity of each pair of plane candidate regions is calculated by the Euclidean distance, and a similarity matrix is ​​constructed based on the similarity calculation results. Each element of the matrix represents the similarity between two plane candidate regions. Each plane candidate region is used as a leaf node of a binary tree. Initially, each plane candidate region is independent. The two plane candidate regions with the highest similarity are selected to merge and create a new parent node. The two plane candidate regions are used as child nodes of the new node and are removed from the binary tree. The similarity formula is as follows:

[0054] Similarity(newNode,P)=max(Similarity(A,P),Similarity(B,P))

[0055] Where ewNode is the created parent node, A and B are the merged plane candidate regions, and P represents other plane candidate regions. During the update process, it is necessary to remove the similarity associated with the merged plane candidate regions A and B, that is: RemoveSimilarity(A,B), and delete the similarity between A and B from the similarity matrix. Update the similarity matrix between the merged plane candidate region and other candidate regions. The updated similarity matrix will contain the similarities of other plane candidate regions. Repeat the merging and updating until only one plane candidate region remains or the preset number of clusters is reached. Through such an iterative process, a binary tree structure can be constructed, in which each node represents a plane candidate region, and the connection between nodes represents the merging relationship of the plane candidate regions. The final clustering result can be obtained by traversing the binary tree, and the root node is the final foreground segmentation result. The results after hierarchical clustering are expanded and eroded to further optimize the segmentation results, remove noise or fill holes.

[0056] The specific operation steps and operation principles of the real-time reconstruction method of the ship comprehensive test scenario based on Bundle Fusion in this embodiment can refer to the operation steps and operation principles of the real-time reconstruction method of the ship comprehensive test scenario based on Bundle Fusion in this embodiment.

[0057] The present invention adopts the real-time three-dimensional fusion algorithm of Bundle Fusion, which can quickly process data from RGBD cameras and monocular cameras to achieve real-time fusion and reconstruction of data. The algorithm optimizes the data processing flow, reduces calculation delays, and improves the real-time performance of reconstruction. In view of the difficulty of multi-view data fusion, the present invention proposes a local optimization and global optimization strategy. By considering the geometric constraints and feature matching between different perspectives, accurate alignment and fusion of data are achieved, reducing errors and omissions. In order to reduce the requirements for computing resources, the present invention adopts an efficient CPU parallel processing strategy. By optimizing the algorithm structure and parallel processing technology, the present invention can reduce the demand for computing resources while ensuring the accuracy of reconstruction and improve the overall performance of the system.

[0058] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A real-time 3D fusion reconstruction method for a ship comprehensive test scene, characterized in that include: Read the internal parameter information of the RGBD camera and the monocular camera, collect the target image data stream of the ship test scene based on the RGBD camera and the monocular camera, pre-process the target image data stream to convert the image coordinate system to the camera coordinate system and obtain a sparse point cloud model to construct a voxel three-dimensional grid. Use Gaussian difference pyramid to construct scale space to detect key points of images at different scales, calculate the gradient direction histogram around the key points and use it as a descriptor, use Euclidean distance to match the descriptor, estimate the relative pose transformation between images to complete the local matching process, project the local frame block to the reference frame to form a key frame, and enter the global matching process: including using SIFT algorithm to detect common feature points between key frames and estimate camera pose, iteratively optimize the camera pose and point cloud position based on Gauss-Newton algorithm and minimize the reprojection error, update the pose information of the key frame according to the optimized camera pose and point cloud position, and fuse the voxel values ​​in the TSDF function to form a dense volumetric grid; The Marching Cubes algorithm is used to extract the surface mesh of the three-dimensional model from the dense volume mesh. Each cube unit in the surface mesh is traversed, and the distance information from the vertex to the nearest surface is obtained based on the cube unit, thereby generating the final three-dimensional model of the test target.

2. The method for real-time three-dimensional fusion reconstruction of a ship comprehensive test scene according to claim 1 is characterized by: Local matching of the target image data stream includes: extracting SIFT feature points from the depth image based on the SIFT algorithm, finding key points in the scale space in the image, determining the position and scale of the SIFT feature points by detecting stable feature points in the image, constructing the scale space using a Gaussian difference pyramid, detecting key points in each scale space, determining key points by finding local extreme points, calculating feature descriptors for the neighborhood area of ​​each key point, generating a gradient direction histogram near the key point to describe the image information around the key point, generating a descriptor vector using the image gradient direction and intensity information near the key point, matching the feature descriptor vectors in the two images, finding the corresponding feature point pairs in different images, measuring the similarity between the two feature vectors using the Euclidean distance, selecting the best matching point pair using the nearest neighbor matching method, estimating the relative pose transformation between the two images through the matched feature point pairs, iteratively selecting a random sample set and calculating the pose transformation model, evaluating the degree of conformity of all matching point pairs with the sparse point cloud model, and selecting the model that minimizes the error between the point clouds as the optimal pose transformation model.

3. The method for real-time three-dimensional fusion reconstruction of a ship comprehensive test scene according to claim 1 is characterized by: The following method is used to form a dense volumetric mesh: feature point matching and pose estimation are performed on the image keyframes based on the SIFT algorithm, and the camera pose and point cloud position are iteratively optimized based on the Gauss-Newton algorithm. In each iteration, the Jacobian matrix J and the residual vector r are calculated, and the Hessian matrix is ​​calculated. The reprojection error function is defined as: E(p)=∑w*||p′-p|| 2 Where p′ is the predicted feature point coordinate, p is the observed feature point coordinate, and w is the weight. The parameters of the pose and point cloud position are iteratively updated in the form of minimizing the reprojection error until the convergence condition is reached, thereby reducing the difference between the projection position and the actual observed image coordinates. The pose information of the key frame is updated according to the optimized pose and point cloud position, and the voxel values ​​in the TSDF function are fused to form a dense volumetric grid. When the TSDF value and weight reach a certain threshold, the voxel is considered to have been stabilized and fused into the global model.

4. The method for real-time three-dimensional fusion reconstruction of a ship comprehensive test scene according to claim 3 is characterized by: When extracting the surface mesh of a three-dimensional model from a dense volumetric mesh: first define an empty triangular mesh data structure to store the generated triangular facets, traverse each voxel unit of the voxel three-dimensional grid, determine the triangular facets that need to be generated on the voxel boundary based on the combination of internal and external labels of the voxel, use the table lookup method for acceleration and optimization, and finally obtain the surface mesh.

5. A method for real-time three-dimensional fusion reconstruction of a ship comprehensive test scene according to any one of claims 1 to 3, characterized in that: When reading the internal parameters of the RGBD camera and the monocular camera, perform lens distortion correction on the camera and use the distortion parameters for dedistortion processing; The preprocessing of the target image data stream includes: setting the filter window size, calculating the spatial distance and grayscale difference between each pixel in each filter window and the central pixel, obtaining the spatial weight and grayscale weight, obtaining the comprehensive weight based on the product of the spatial weight and the grayscale weight, performing weighted averaging on the pixels in the filter window according to the comprehensive weight to obtain the filtered pixel value, sliding the filter window on the image to cover the entire image, and then performing average pooling operation on the image information after bilateral filtering.

Images