Ship small assembly workpiece model visual matching method and device

Abstract

A ship small assembly workpiece model visual matching method and device, comprising: collecting the point cloud data of the top of the ship small assembly workpiece; constructing a three-dimensional model database containing a plurality of ship small assembly workpiece CAD models, and performing surface sampling on each CAD model to generate feature point clouds; collecting workpiece images from multiple angles, performing feature extraction and cross-view fusion on the multi-view images, back-projecting the fused features to generate supplementary point clouds, and splicing the supplementary point clouds with the top point clouds to form dense fusion point clouds; constructing a multi-modal feature matching network based on an attention mechanism, performing cross-modal attention fusion on the geometric features of the dense fusion point clouds and the multi-view fusion image features to generate multi-modal fusion features, performing cross-attention matching on the multi-modal fusion features and the model database feature point clouds, and outputting a workpiece category recognition result and a pose transformation matrix; the present application realizes deep fusion of multi-view multi-modal data, improves matching accuracy and robustness, and is suitable for high-precision visual matching under complex working conditions of shipbuilding.

Description

Technical Field

[0001] This invention relates to the field of shipbuilding technology, and specifically to a visual matching method and apparatus for ship sub-assembly workpiece models. Background Technology

[0002] In the process of ship hull construction, small assembly workpieces are small modular structural components made up of basic components such as steel plates and profiles through cutting, blanking, assembly, and spot welding. The accurate identification and posture matching of small assembly workpieces are key links to realize automated welding and intelligent assembly, which directly affects the efficiency and quality of ship construction.

[0003] In existing technologies, visual inspection and 3D matching of ship assembly workpieces generally employ single-modal sensing devices paired with fixed single-view vision cameras to complete data acquisition.

[0004] For example, point cloud matching-based schemes acquire workpiece point cloud data through 3D laser scanning and then register it with a theoretical model; however, such methods can only extract shallow features in the spatial domain within a fixed field of view and cannot fully acquire the workpiece's overall structural information. Image fusion-based detection schemes improve detection performance by fusing visible light and infrared images; however, these methods are limited to the two-dimensional image level and do not achieve three-dimensional spatial feature fusion, making it difficult to meet the requirements of three-dimensional matching.

[0005] The existing technical solutions have obvious defects: Single-view acquisition mode is easily affected by factors such as on-site occlusion, shooting angle limitation, and surface reflection of objects, resulting in a large number of missing areas in the 3D point cloud data. Furthermore, it lacks mature point cloud completion and accurate alignment methods, making it difficult to construct a complete and standardized 3D feature model of the workpiece. The system failed to achieve multi-view data linkage and fusion, resulting in the output of fragmented and one-sided single-view point cloud data with insufficient global feature representation capabilities. Traditional feature matching methods do not distinguish the importance of features, cannot select effective core features, are easily affected by environmental noise and redundant information, and frequently result in mismatches during workpiece model matching, leading to low matching accuracy and operational stability. Summary of the Invention

[0006] This invention proposes a visual matching method and device for ship assembly workpiece models, which solves the problems of incomplete data, fragmented features, and low matching accuracy caused by single-view acquisition in the prior art, and achieves global, high-precision, stable and reliable visual matching.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows: A visual matching method for ship assembly workpiece models includes: Step 1) Collect point cloud data directly above the workpiece erected by the ship crew; Step 2) Construct a 3D model database containing CAD models of various ship assembly parts, and perform surface sampling on the CAD model of each part in the database to generate feature point cloud data with uniform distribution. Step 3) Acquire workpiece images from multiple angles using a panoramic camera, extract features from the multi-view images and fuse them across different perspectives, back-project the fused image features onto a three-dimensional space to generate a supplementary point cloud, and stitch the supplementary point cloud with the point cloud data directly above to form a dense fused point cloud. Step 4) Construct a multimodal feature matching network based on an attention mechanism, and perform cross-modal attention fusion of the geometric features of the dense fused point cloud with the multi-view fused image features to generate multimodal fused features; The multimodal fusion features are cross-attention matched with the feature point cloud in the model database to output the workpiece category recognition result and the six-degree-of-freedom pose transformation matrix.

[0008] Further, step 1 includes Step 101) Based on the 3D depth camera deployed directly above the workpiece, acquire the original point cloud data of the ship group's assembled workpiece; Step 102) Preprocess the raw point cloud data, including outlier filtering, regional downsampling, and region of interest (ROI) extraction.

[0009] Further, in step 102, the regional downsampling includes Downsampling is performed on the workpiece base plate area and non-base plate area separately, with a larger downsampling rate for the base plate area and a smaller downsampling rate for the non-base plate area.

[0010] Further, in step 2, surface sampling is performed on the CAD model of each workpiece in the database to generate feature point cloud data, including... Step 201) Perform multi-resolution surface sampling on the CAD model of the small-scale workpiece; Step 202) Adaptive density adjustment is performed based on local surface curvature, increasing the sampling density in the edge, corner and weld areas to generate feature point cloud data with uniform distribution.

[0011] Furthermore, in step 3, feature extraction and cross-view fusion are performed on the multi-view images, including... Step 301) Perform spatiotemporal alignment on the multi-view image data; Step 302) Based on the common features of the multi-view images, perform fusion to supplement and repair the point cloud data above.

[0012] A visual matching device for ship sub-assembly workpiece models, characterized in that it executes the visual matching method for ship sub-assembly workpiece models, including... The point cloud acquisition module is used to acquire point cloud data directly above the workpiece and perform preprocessing. The model library building module is used to store CAD models of ship sub-assemblies and sample and generate feature point cloud data; The multi-view fusion module is used to acquire workpiece images from multiple angles, fuse image features and generate supplementary point clouds, which are then stitched together with the point cloud data directly above the workpiece to obtain a dense fused point cloud. The multimodal matching module performs cross-modal attention fusion of the geometric features of the dense fused point cloud with the fused image features to generate multimodal fused features. The multimodal fused features are then cross-attention matched with the feature point cloud in the model database to output the workpiece category recognition result and the six-degree-of-freedom pose transformation matrix.

[0013] Furthermore, the point cloud acquisition module includes a 3D depth camera acquisition unit and a preprocessing unit, wherein the preprocessing unit performs outlier filtering, regional downsampling, and ROI extraction.

[0014] Furthermore, the multimodal matching module includes a cross-modal fusion unit and a cross-attention matching unit. The cross-modal fusion unit obtains multimodal fusion features by fusing point cloud geometric features and image features through cross-modal attention weighted fusion. The cross-attention matching unit outputs the workpiece category recognition result and a six-degree-of-freedom pose transformation matrix based on the cross-attention comparison of the multimodal fusion features and the feature point cloud in the model database.

[0015] Compared with the prior art, the present invention has the following advantages: By employing multi-view panoramic acquisition combined with multi-modal sensor data, the problem of incomplete data caused by blind spots or obstructions in observation is solved; By generating supplementary point clouds through back projection of multi-view image features, missing point cloud completion and spatial alignment are achieved, constructing complete three-dimensional features of the workpiece, and replacing the traditional single-view fragmented point cloud output. By using an attention mechanism model, we can achieve deep fusion of features from multiple perspectives, strengthen key features, suppress noise interference, reduce false matching rate, and improve matching accuracy and robustness. In response to the complex structure and strong interference of ship assembly workpieces, a global, high-precision, stable and reliable vision matching solution has been developed, which meets the actual needs of industrial sites. Attached Figure Description

[0016] Figure 1 This is a flowchart of an embodiment of the present invention; Figure 2a A top view of the environment configuration for an embodiment of the present invention; Figure 2b Side view of the environment configuration for an embodiment of the present invention; Figure 3 This is a diagram of the surrounding image fusion network architecture according to an embodiment of the present invention; Figure 4 This is a diagram of the cross-modal matching network architecture according to an embodiment of the present invention. Detailed Implementation

[0017] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] This embodiment proposes a visual matching method for ship assembly workpiece models. (See reference...) Figure 1 ,include: Step S1) Collect raw point cloud data of the ship group's assembled workpiece based on a 3D depth camera deployed directly above the workpiece; As shown in Figure 2, the matching environment configuration diagram of this embodiment of the invention is shown, wherein a 3D depth camera is deployed directly above the workpiece to acquire top-view 3D point cloud data of the workpiece. The raw point cloud data is preprocessed, including outlier filtering, regional downsampling, and region of interest (ROI) extraction. Outlier filtering is used to remove noise points and outliers in point cloud data to improve data quality. Specifically, a statistical outlier removal algorithm can be used to calculate the average distance from each point to its nearest neighbor and remove points whose distance exceeds a threshold. Regional downsampling is used to reduce the amount of point cloud data while retaining the feature information of key areas. Specifically, the workpiece base plate area and non-base plate area are downsampled separately, with a larger downsampling rate for the base plate area and a smaller downsampling rate for the non-base plate area. Region of Interest (ROI) extraction is used to cut out the effective region containing the workpiece from the original point cloud, and remove the background and irrelevant regions based on the preset workpiece size range and position information.

[0019] Step S2) Construct a 3D model database containing CAD models of various ship assembly parts, and perform surface sampling on the CAD model of each part in the database to generate feature point cloud data; First, CAD models of various typical ship assembly parts were collected, including assembly parts of different specifications and structural forms, and a three-dimensional model database was constructed. Then, multi-resolution surface sampling is performed on the CAD model of each workpiece in the database to generate feature point cloud data with uniform distribution.

[0020] In this embodiment, PoissonDisk sampling is performed on the CAD model of the small assembled workpiece, and adaptive density adjustment is performed based on the local surface curvature. The sampling density is increased in the edge, corner and weld areas to generate feature point cloud data with uniform distribution and rich key regional features.

[0021] Step S3) Collect workpiece images from multiple angles using a panoramic camera, extract features from the multi-view images and fuse them across different perspectives, and back-project the fused image features onto a three-dimensional space to generate a supplementary point cloud. Then, stitch the supplementary point cloud with the point cloud data above to form a dense fused point cloud. Referring to Figure 2, multiple vision cameras are deployed around the workpiece to capture images of the workpiece from different angles. In this embodiment, four surround-view cameras are deployed, located in the front, back, left and right directions of the workpiece, respectively, and distributed at a 90-degree angle to the center of the workpiece. The height of the cameras is the same as the height of the center of the workpiece.

[0022] For the specific process of feature extraction and cross-view fusion of multi-view images, please refer to Figure 3 Feature extraction is performed on images acquired from each perspective to obtain feature maps for each perspective; spatiotemporal alignment is performed on multi-view image data to ensure consistency of images from different perspectives in time and space; fusion is performed based on the common features of multi-view images to supplement and repair the point cloud data above.

[0023] Based on the camera calibration parameters, the feature points of the fused two-dimensional image are back-projected into three-dimensional space to generate three-dimensional supplementary point cloud data. This supplementary point cloud can make up for the data loss of the upper point cloud on the side and in the occluded area. The supplementary point cloud is then stitched and fused with the point cloud data directly above the workpiece to form a dense fused point cloud covering the surface of the workpiece.

[0024] Step S4) Construct a multimodal feature matching network based on an attention mechanism, perform cross-modal attention fusion of the geometric features of the dense fused point cloud with the multi-view fused image features to generate multimodal fused features; perform cross-attention matching of the multimodal fused features with the feature point cloud in the model database to output the workpiece category recognition result and the six-degree-of-freedom pose transformation matrix.

[0025] See Figure 4 The geometric features of the dense fused point cloud are fused with the multi-view fused image features through cross-modal attention fusion to generate multimodal fused features. Then, the multimodal fused features are cross-attention matched with the feature point cloud in the model database.

[0026] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A visual matching method for ship assembly workpiece models, characterized in that, include: Step 1) Collect point cloud data directly above the workpiece erected by the ship crew; Step 2) Construct a 3D model database containing CAD models of various ship assembly parts, and perform surface sampling on the CAD model of each part in the database to generate feature point cloud data with uniform distribution. Step 3) Acquire workpiece images from multiple angles using a panoramic camera, extract features from the multi-view images and fuse them across different perspectives, back-project the fused image features onto a three-dimensional space to generate a supplementary point cloud, and stitch the supplementary point cloud with the point cloud data directly above to form a dense fused point cloud. Step 4) Construct a multimodal feature matching network based on an attention mechanism, and perform cross-modal attention fusion of the geometric features of the dense fused point cloud with the multi-view fused image features to generate multimodal fused features; The multimodal fusion features are cross-attention matched with the feature point cloud in the model database to output the workpiece category recognition result and the six-degree-of-freedom pose transformation matrix.

2. The visual matching method for ship sub-assembly workpiece models according to claim 1, characterized in that, Step 1 includes Step 101) Based on the 3D depth camera deployed directly above the workpiece, acquire the original point cloud data of the ship group's assembled workpiece; Step 102) Preprocess the raw point cloud data, including outlier filtering, regional downsampling, and region of interest (ROI) extraction.

3. The visual matching method for ship sub-assembly workpiece models according to claim 2, characterized in that, In step 102, the regional downsampling includes Downsampling is performed on the workpiece base plate area and the non-base plate area separately, with a larger downsampling rate for the base plate area and a smaller downsampling rate for the non-base plate area.

4. The visual matching method for ship sub-assembly workpiece models according to claim 2, characterized in that, In step 2, surface sampling is performed on the CAD model of each workpiece in the database to generate feature point cloud data, including... Step 201) Perform multi-resolution surface sampling on the CAD model of the small-scale workpiece; Step 202) Adaptive density adjustment is performed based on local surface curvature, increasing the sampling density in the edge, corner and weld areas to generate feature point cloud data with uniform distribution.

5. The visual matching method for ship sub-assembly workpiece models according to claim 1, characterized in that, In step 3, feature extraction and cross-view fusion are performed on the multi-view images, including... Step 301) Perform spatiotemporal alignment on the multi-view image data; Step 302) Based on the common features of the multi-view images, perform fusion to supplement and repair the point cloud data above.

6. A visual matching device for a ship assembly workpiece model, characterized in that, The visual matching method for ship sub-assembly workpiece models as described in claim 1 includes: The point cloud acquisition module is used to acquire point cloud data directly above the workpiece and perform preprocessing. The model library building module is used to store CAD models of ship sub-assemblies and sample and generate feature point cloud data; The multi-view fusion module is used to acquire workpiece images from multiple angles, fuse image features and generate supplementary point clouds, which are then stitched together with the point cloud data directly above the workpiece to obtain a dense fused point cloud. The multimodal matching module performs cross-modal attention fusion of the geometric features of the dense fused point cloud with the fused image features to generate multimodal fused features. The multimodal fused features are then cross-attention matched with the feature point cloud in the model database to output the workpiece category recognition result and the six-degree-of-freedom pose transformation matrix.

7. The visual matching device for ship assembly workpiece models according to claim 6, characterized in that, The point cloud acquisition module includes a 3D depth camera acquisition unit and a preprocessing unit. The preprocessing unit performs outlier filtering, regional downsampling, and ROI extraction.

8. The visual matching device for ship assembly workpiece models according to claim 6, characterized in that, The multimodal matching module includes a cross-modal fusion unit and a cross-attention matching unit. The cross-modal fusion unit obtains multimodal fusion features by fusing point cloud geometric features and image features through cross-modal attention weighted fusion. The cross-attention matching unit outputs the workpiece category recognition result and a six-degree-of-freedom pose transformation matrix based on the cross-attention comparison of the multimodal fusion features and the feature point cloud in the model database.

Images