Hull defect detection system based on machine vision

Abstract

The invention provides a ship body defect detection system based on machine vision, and aims to realize automatic and high-precision detection of ship body surface defects, the system adopts a ZED binocular camera to acquire real-time images of a ship body, performs three-dimensional reconstruction of the ship body through an ORB-SLAM framework, and realizes accurate registration of point clouds in combination with an SAC-IA rough matching algorithm and an ICP fine matching algorithm. The defect detection system utilizes the deep learning network Faster RCNN to classify and position the detected defects, generates a defect report and an alarm signal through an automatic defect classification system, has the characteristics of high efficiency, automation and high detection precision, can realize real-time monitoring and alarm of hull surface defects in a complex environment, and has a wide application prospect. The method supports multi-sensor data fusion, has good expansibility, is suitable for defect detection of large-area and complex structures of ship bodies, remarkably improves the detection efficiency, reduces the risks of missing detection and false alarm, and provides powerful guarantee for safe operation of ships.

Description

Technical Field

[0001] The present invention relates to the technical field of specific industrial applications of artificial intelligence, and in particular to a hull defect detection system based on machine vision. Background Art

[0002] Ships, as the main means of maritime transportation, are an important part of global logistics, national defense construction and marine development. As ships are exposed to the marine environment for a long time, the hull is often affected by external factors such as corrosion, friction, and collision during use, which easily leads to structural defects such as cracks, holes, wear, and dents. If these defects are not discovered and repaired in time, they may cause serious safety hazards, even threaten the lives of crew members, and cause significant economic losses. Therefore, early detection and maintenance of hull defects are the key to ensuring the safe operation of ships.

[0003] Limitations of traditional hull inspection methods. At present, hull defect detection mainly relies on manual inspection combined with instruments, such as ultrasonic flaw detectors, magnetic particle flaw detectors, eddy current detectors, etc. Although these methods can effectively detect some defects, they have the following limitations:

[0004] Strong reliance on manual operation: Traditional methods mostly require inspectors to check the hull one by one at specific locations. Due to the complex structure of the ship and the huge area of ​​the hull, manual inspection is inefficient and easily affected by human factors, which can easily lead to missed inspections and misjudgments.

[0005] Time-consuming and costly: Inspections of large ships usually require docking or dry docking, which is not only time-consuming but also leads to high downtime costs and inspection fees.

[0006] Limited detection range: Traditional detection methods have low sensitivity for detecting small surface defects (such as tiny cracks, holes, etc.), especially for defects on complex surfaces and internal structures, which are difficult to accurately identify.

[0007] Poor real-time performance: Since traditional inspection relies on manual inspection or instrument detection, it is impossible to achieve real-time monitoring of hull defects. Defects may deteriorate rapidly in a short period of time and cannot be discovered and handled in time.

[0008] Widespread application of machine vision technology and its introduction in marine equipment. In recent years, with the development of computer vision and deep learning technology, automated inspection technology based on machine vision has been widely used. Machine vision technology collects image data of the hull through high-precision cameras, sensors and other equipment, and analyzes the defect characteristics in the image through algorithms, which can effectively make up for the shortcomings of traditional methods. Especially in the marine field, machine vision technology has the following advantages:

[0009] High efficiency and automation: The machine vision system can realize fully automatic, all-weather real-time monitoring of the hull, avoiding manual dependence and human errors in traditional detection methods.

[0010] High precision and high coverage: Based on image processing and deep learning technology, machine vision can detect tiny defects at the micron level and cover all areas of the hull, including complex structures and curved surfaces that are difficult to reach.

[0011] Real-time monitoring and alarm: The machine vision system can monitor the status of the hull in real time and automatically trigger an alarm when defects are detected, notifying relevant personnel to make timely repairs, greatly reducing the safety hazards caused by hull defects.

[0012] Scalability: The machine vision system can be combined with other sensors (such as laser scanners, infrared thermal imaging, etc.) to further improve the accuracy of detection and achieve comprehensive analysis of internal and external defects of the hull.

[0013] Although some studies and applications have introduced machine vision into hull defect detection, the existing technology still has the following challenges and problems: (1) The problem of three-dimensional reconstruction accuracy. Some current systems rely on two-dimensional images for defect detection and lack accurate modeling of the three-dimensional structure of the hull, resulting in low accuracy in defect positioning on complex surfaces. How to achieve high-precision reconstruction of the three-dimensional structure of the hull based on binocular vision technology is an urgent problem to be solved. (2) The problem of point cloud matching accuracy and speed. The generation and matching of three-dimensional point clouds are key steps based on machine vision. The matching accuracy and speed of existing methods in the processing of large-scale point cloud data still cannot meet the actual application requirements, especially in real-time detection and large-area detection tasks. How to effectively improve the accuracy and efficiency of point cloud matching has become a difficulty. (3) The problem of robustness and false alarm rate of defect detection: Existing defect detection systems are prone to detection errors in complex marine environments (such as those affected by light, humidity, salt corrosion, etc.), and there are false alarms or missed alarms. The robustness and accuracy of the system need to be improved urgently. (4) Insufficient system integration and intelligence: Most existing machine vision inspection systems have single functions and are difficult to achieve multimodal data fusion, intelligent analysis and early warning, and cannot meet the comprehensive and multi-type defect detection needs of complex ship structures.

[0014] To this end, we designed a hull defect detection system based on machine vision to solve the above problems. Summary of the invention

[0015] The purpose of the present invention is to provide a hull defect detection system based on machine vision to address the deficiencies of the prior art. The hull defect detection system based on machine vision of the present invention is required to reduce the investment of personnel costs, fully cover the designated inspection content, have a model classification accuracy of more than 98%, and complete the inspection process fully automatically and efficiently, with a high detection rate and a low false alarm rate.

[0016] To achieve the above object, the present invention provides the following technical solutions:

[0017] A hull defect detection system based on machine vision, the hull defect detection system includes a ZED binocular camera, an ORB-SLAM framework, a point cloud matching module, a defect detection module, a target recognition module and an automatic defect classification system, wherein:

[0018] ZED binocular camera: used to obtain real-time stereo image data of the hull and generate a three-dimensional point cloud of the hull through depth information;

[0019] ORB-SLAM framework: used to perform 3D reconstruction of the ship structure through visual mileage calculation method, achieving accurate spatial positioning and reconstruction model;

[0020] Point cloud matching module: used to perform preliminary alignment through the SAC-IA coarse matching algorithm, and use the ICP fine matching algorithm to perform high-precision registration of the source point cloud and the target point cloud;

[0021] Defect detection module: By setting a distance threshold to compare the difference between the source point cloud and the target point cloud, the defect area on the hull surface can be identified and located;

[0022] Target recognition module: uses the FasterRCNN network based on deep learning to classify and accurately locate the detected defects;

[0023] Automatic Defect Classification System (ADCS): Uploads the identified defect information to the system monitoring and early warning platform, generates alarm signals and provides real-time feedback.

[0024] As a preferred technical solution of the present invention, the SAC-IA rough matching algorithm performs a random sampling consistency initialization on the point cloud and realizes the rough matching of the point cloud based on the following optimization model:

[0025]

[0026] Among them, R is the rotation matrix, t is the translation vector, and p i is a point in the source point cloud, q i is the corresponding point in the target point cloud, E SAC-IA is the error function, and the coarse matching optimizes the initial alignment of the point cloud by reducing the error.

[0027] As a preferred technical solution of the present invention, the ICP fine matching algorithm is used to further optimize the point cloud after rough matching, and the registration is performed by minimizing the mean square error between the source point cloud and the target point cloud. The optimization process is based on the following formula:

[0028]

[0029] Among them, M is the number of point pairs selected in fine matching, p i is a point in the source point cloud, q i is the corresponding point in the target point cloud, R is the rotation matrix, t is the translation vector, and accurate point cloud registration is achieved by minimizing the mean square error through multiple iterative calculations.

[0030] As a preferred technical solution of the present invention, the defect detection module performs detection through the geometric form of the three-dimensional point cloud. The distance threshold δ set in the detection process is used to identify points that are obviously deviated from the normal hull surface, and defect judgment is performed based on the following formula:

[0031] d=∥p i -q i ∥

[0032] Among them, d is the Euclidean distance between the source point cloud and the target point cloud. When d>δ, the point is considered to have a potential defect.

[0033] As a preferred technical solution of the present invention, the target recognition module adopts the deep learning network FasterRCNN, wherein:

[0034] The Region Proposal Network (RPN) generates multiple candidate regions of interest (ROIs) that may contain defects;

[0035] Use convolutional neural network to extract features from the extracted region of interest. The network extracts deep features of the defect area through multiple convolution operations.

[0036] Finally, the extracted features are classified and bounding box regressed through the fully connected layer to output the defect category (such as cracks, holes, wear, etc.) and its specific location coordinates.

[0037] As the preferred technical solution of the present invention, the system supports the conversion of multiple point cloud file formats, including OBJ, STL, PLY and PCD formats, to facilitate compatible processing of different data sources. The specific format conversion uses VTK (Visualization Toolkit) to write a conversion algorithm to convert other formats into PCD format to facilitate subsequent point cloud processing.

[0038] As a preferred technical solution of the present invention, the system uses the Kdtree data structure to accelerate the search for nearest neighbors in three-dimensional point clouds, and finds the nearest neighbor points of the source point cloud and the target point cloud during the matching process through a fast neighbor search algorithm, thereby reducing calculation time and improving the efficiency and accuracy of defect detection.

[0039] As a preferred technical solution of the present invention, the automatic defect classification system can automatically generate a classification model library according to different types of hull defects (such as corrosion, dents, cracks, etc.). The system can regularly update and optimize these models to cope with various defect detection needs under different hull materials, ship types and environments.

[0040] As a preferred technical solution of the present invention, the system supports timed detection and manual trigger detection modes. Users can set the system to start the detection process at a fixed time as needed, and monitor the image data in real time through the ship video network. The timed detection mode can automatically perform periodic detection without human intervention, and the manual mode can be started by the operator at any time.

[0041] As a preferred technical solution of the present invention, the system can automatically generate a defect detection report based on the detection results. The report includes the following contents:

[0042] The location coordinates of the defect (X, Y, Z);

[0043] Defect type classification (such as holes, cracks, wear, etc.);

[0044] The severity of the defect (minor, moderate, severe);

[0045] System-generated repair suggestions;

[0046] Historical defect records and their changing trend analysis reports to facilitate subsequent maintenance decisions.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] 1. The present invention realizes the three-dimensional reconstruction of the hull structure through the ZED binocular camera and ORB-SLAM framework, and combines the SAC-IA coarse matching and ICP fine matching algorithms to accurately align the point cloud, which can automatically and efficiently perform comprehensive detection of hull surface defects (such as cracks, holes, wear, etc.). Compared with traditional detection methods that rely on manual and instrumental methods, this system significantly reduces detection time, improves detection coverage, and reduces labor costs.

[0049] 2. Based on machine vision technology and the deep learning network Faster RCNN, the present invention can not only accurately identify micron-level small defects, but also classify them according to different defect types (such as cracks, holes, etc.). Through the improved point cloud matching algorithm and deep learning technology, the system has extremely high defect detection accuracy and robustness, and can achieve low false alarms and missed alarms in complex environments (such as humidity, corrosion, light changes, etc.).

[0050] 3. The system uses the Automatic Defect Classification System (ADCS) to achieve real-time detection and classification of defects, automatically generate defect reports, and trigger alarms in a timely manner when serious defects are detected, notifying relevant maintenance personnel to handle them. At the same time, the system supports the fusion of multiple sensor data, has strong scalability, can adapt to a variety of ship environments and complex defect detection tasks, and provides intelligent protection for safe ship operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a structural block diagram of a hull defect detection system based on machine vision proposed by the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] The present invention first designs a defect detection scheme based on a point cloud matching algorithm and a defect detection scheme based on deep learning. Based on the ZED binocular camera and the open source ORB-SLAM project, the real-time 3D reconstruction of the ZED camera is realized. In order to unify the model data into the standard point cloud format PCD of PCL (PointCloud Library) for easy detection, the conversion algorithm between OBJ, STL, PLY format and PCD format is written and improved using the 3D vision library VTK (visualization toolkit). In order to obtain better point cloud matching results, the matching is divided into two steps. First, the SAC-IA (Sample Consensus Initial Aligment) algorithm is used to perform a rough match on the point cloud. The rough match can provide a better initialization result for the next step of fine matching. Then, the ICP (Iterative Cloest Point) algorithm is used to achieve fine matching of the point cloud. The fast approximate neighbor search algorithm based on the Kdtree data structure is used to find the defect location on the fine matching result.

[0054] The workflow of the system is as follows: connect the system to the integrated video network of the whole ship, start the system monitoring and early warning platform and automatically start the built-in automatic defect classification system (Auto Defect Classify System, ADCS for short), adopt manual or system timing detection mode, based on the image transmitted from the video network, the system divides the image into pictures, saves it to the database and performs feature capture. The ADCS system identifies and locates defects by matching the model library, and sends the defect information to the system monitoring and early warning platform for alarm prompts.

[0055] The following is combined with Figure 1 and multiple embodiments, the specific implementation methods of the present invention are further described in detail.

[0056] Example 1: 3D reconstruction and defect detection of ship hull based on machine vision

[0057] This example describes how to use the ZED binocular camera, ORB-SLAM framework, SAC-IA and ICP algorithms to achieve 3D reconstruction and defect detection of the ship hull.

[0058] 1. Image acquisition and 3D reconstruction: ZED binocular cameras are installed at multiple key locations around the hull to collect left and right perspective images of the hull. Through the images obtained by the binocular camera, the system can generate a depth map in real time and convert it into 3D point cloud data.

[0059] The ORB-SLAM framework is responsible for processing the collected image data. The system uses visual odometry to detect feature points in the image, track the feature points, calculate the camera's posture and position, and thus achieve real-time reconstruction of the 3D structure of the ship.

[0060] 2. Point cloud matching and registration SAC-IA rough matching: In order to preliminarily align the source point cloud and target point cloud of the hull, the system uses the SAC-IA (Sample Consensus Initial Alignment) algorithm. This algorithm selects the initial matching point pair by random sampling consistency method. Optimization goal:

[0061]

[0062] Among them, R is the rotation matrix, t is the translation vector, and p i and q i are the points in the source point cloud and the target point cloud respectively, and N is the number of point pairs involved in coarse matching.

[0063] ICP fine matching: After the rough matching is completed, the system further uses the ICP (Iterative Closest Point) algorithm for fine matching. The ICP algorithm iteratively optimizes the point pair error between the source point cloud and the target point cloud to ensure high-precision alignment.

[0064] Error minimization formula:

[0065]

[0066] Where M is the number of point pairs in the fine matching, p i and q i are points in the source and target point clouds.

[0067] 3. Defect detection and distance threshold determination: After the registration is completed, the system calculates the distance d between each corresponding point of the source point cloud and the target point cloud to determine whether there are defects on the hull surface:

[0068] If d exceeds the preset threshold δ, the point is considered to be a defect point.

[0069] Distance formula:

[0070] d=∥p i -q i ∥

[0071] When d>δ, the system will mark the point as a defect and further analyze the nature and severity of the defect.

[0072] Example 2: Hull defect identification based on deep learning

[0073] This embodiment focuses on describing how to identify and classify defects on the hull surface through the FasterRCNN deep learning network.

[0074] 1. Image preprocessing and feature extraction: Image acquisition: The images collected by the ZED binocular camera are preprocessed to remove noise and enhance image contrast and brightness to ensure that subsequent feature extraction is more accurate.

[0075] ROI (Region of Interest) Extraction: The system uses the Region Proposal Network (RPN) in FasterRCNN to generate multiple candidate regions that may contain hull defects (such as cracks, holes, wear, etc.).

[0076] 2. Defect classification and location: Feature extraction and convolution operation: The system uses convolutional neural network (CNN) to extract features from the extracted ROI area, and multiple convolution layers perform deep learning analysis on potential defects in the image.

[0077] Defect classification and bounding box regression: Through the fully connected layer, the system classifies the extracted features, generates the corresponding defect type (such as cracks, holes, and wear), and regresses the bounding box of the defect to obtain the precise defect location.

[0078] 3. Output results and alarms: The system generates a report containing information such as defect location, size, and category based on the identified defect category, and uploads the report to the monitoring and early warning platform. If the defect exceeds the set severity, the system will automatically trigger an alarm and notify relevant personnel to handle it.

[0079] Example 3: Hull detection based on multimodal data fusion

[0080] This embodiment demonstrates how to accelerate the process of hull detection through conversion of multiple point cloud data formats and fast neighbor search.

[0081] 1. Conversion of multiple point cloud data formats: In order to process multiple 3D models exported from different sensors or software, the system supports the conversion of multiple point cloud file formats, including OBJSTL, PLY and PCD formats. VTK (Visualization Toolkit): used to write data conversion algorithms to ensure that models in different formats can be uniformly converted to PCD format for subsequent point cloud processing.

[0082] 2. Nearest neighbor search acceleration: The system uses the Kdtree data structure to accelerate the nearest neighbor search in the 3D point cloud, so as to quickly find the nearest neighbor point pair of the source point cloud and the target point cloud, thereby optimizing the matching and defect detection process. Nearest neighbor search formula: In the nearest neighbor point search process in the point cloud, the system determines the optimal point pair through the following formula:

[0083]

[0084] Among them, P i is a point in the source point cloud, Q i is a point in the target point cloud, NN(P i ) is P i The corresponding nearest neighbor point.

[0085] 3. Integrate multimodal data: The system can fuse data from different sensors (such as ZED binocular cameras, laser scanners, etc.) to generate high-precision 3D models, and improve the speed and accuracy of point cloud matching and defect detection through nearest neighbor search algorithms.

[0086] Example 4: Real-time hull detection and alarm based on automatic defect classification system (ADCS)

[0087] This embodiment describes how to implement real-time detection, report generation and alarm mechanism of hull defects through the Automatic Defect Classification System (ADCS).

[0088] 1. Automatic detection mode: scheduled detection and real-time monitoring: The system can perform scheduled detection at preset time intervals or real-time detection through manual triggering by the operator. The ZED binocular camera transmits the hull image to the central server, and the system automatically starts the defect detection process according to the detection mode.

[0089] Data transmission and storage: The collected images and test results will be saved in the database. The system will also transmit the defect information to the monitoring platform via FTP or RabbitMQ protocol for real-time viewing and analysis.

[0090] 2. Report generation and alarm mechanism: Defect classification report generation: The system will automatically generate a detailed defect report based on the test results. The report content includes the coordinate location, type, size and severity of the defect.

[0091] Automatic alarm: When the detected defects exceed the set threshold (for example, the width of the crack exceeds 1 mm or the diameter of the hole exceeds 5 mm), the system will automatically trigger an alarm and send the alarm information to the relevant maintenance personnel.

[0092] Practice has proved that the detection system and method of the invention can greatly reduce the investment in personnel costs, cover all aspects of designated inspection content, and achieve a model classification accuracy of more than 98%. The inspection process can be completed fully automatically and efficiently, with the advantages of high detection rate and low false alarm rate.

[0093] The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in the field. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A hull defect detection system based on machine vision, characterized in that: The hull defect detection system includes a ZED binocular camera, an ORB-SLAM framework, a point cloud matching module, a defect detection module, a target recognition module and an automatic defect classification system, among which: ZED binocular camera, used to obtain real-time stereo image data of the hull and generate a three-dimensional point cloud of the hull through depth information; ORB-SLAM framework, used to perform three-dimensional structural reconstruction of the hull through visual mileage calculation method, to achieve spatial positioning and reconstruction model; Point cloud matching module, used to perform preliminary alignment using the SAC-IA coarse matching algorithm, and to perform high-precision registration of the source point cloud and the target point cloud using the ICP fine matching algorithm; Defect detection module, which compares the difference between the source point cloud and the target point cloud by setting a distance threshold, and identifies and locates the defective area on the hull surface; The target recognition module uses the FasterRCNN network based on deep learning to classify and accurately locate the detected defects; The automatic defect classification system uploads the identified defect information to the system monitoring and early warning platform, generates alarm signals and provides real-time feedback.

2. A hull defect detection system based on machine vision according to claim 1, characterized in that: The SAC-IA rough matching algorithm performs a random sampling consistency initialization on the point cloud and realizes the rough matching of the point cloud based on the following optimization model: Among them, R is the rotation matrix, t is the translation vector, and p i is a point in the source point cloud, q i is the corresponding point in the target point cloud, E SAC-IA is the error function.

3. The hull defect detection system based on machine vision according to claim 1 is characterized in that: The ICP fine matching algorithm is used to further optimize the point cloud after rough matching by minimizing the mean square error between the source point cloud and the target point cloud. The optimization process is based on the following formula: Among them, M is the number of point pairs selected in fine matching, p i is a point in the source point cloud, q i is the corresponding point in the target point cloud, R is the rotation matrix, t is the translation vector, and the mean square error is minimized through multiple iterative calculations.

4. The hull defect detection system based on machine vision according to claim 1, characterized in that: The defect detection module performs detection through the geometric form of the three-dimensional point cloud. The distance threshold δ set during the detection process is used to identify points that are significantly deviated from the normal hull surface, and defect judgment is performed based on the following formula: d=∥p i -q i ∥ Among them, d is the Euclidean distance between the source point cloud and the target point cloud. When d>δ, the point is considered to have a potential defect.

5. The hull defect detection system based on machine vision according to claim 1 is characterized in that: The target recognition module adopts the deep learning network FasterRCNN, where: The region proposal network generates multiple candidate regions of interest that may contain defects; Use convolutional neural network to extract features from the extracted region of interest. The network extracts deep features of the defect area through multiple convolution operations. Finally, the extracted features are classified and bounding box regressed through the fully connected layer to output the defect category and its specific location coordinates.

6. The hull defect detection system based on machine vision according to claim 1, characterized in that: The system supports the conversion of multiple point cloud file formats, including OBJ, STL, PLY and PCD formats. The specific format conversion uses VTK to write a conversion algorithm to convert other formats into PCD format.

7. The hull defect detection system based on machine vision according to claim 1 is characterized in that: The system uses the Kdtree data structure to accelerate the search for nearest neighbors in three-dimensional point clouds. It uses a fast nearest neighbor search algorithm to find the nearest neighbor points of the source point cloud and the target point cloud during the matching process, reducing the calculation time.

8. The hull defect detection system based on machine vision according to claim 1, characterized in that: The automatic defect classification system automatically generates a classification model library according to different types of hull defects, and regularly updates and optimizes these models to cope with various defect detection requirements under different hull materials, ship types and environments.

9. The hull defect detection system based on machine vision according to claim 1, characterized in that: The system supports timed detection and manual trigger detection modes. Users can set the system to start the detection process at a fixed time as needed, and monitor image data in real time through the ship's video network. The timed detection mode automatically performs periodic detection without human intervention, and the manual mode can be started by the operator at any time.

10. The hull defect detection system based on machine vision according to claim 1, characterized in that: The system automatically generates a defect detection report based on the detection results. The report includes the following contents: The location coordinates of the defect; Classification of defects by type; The severity of the defect; System-generated repair suggestions; Historical defect records and their change trend analysis report.

Images