New energy ship parts detection method and system based on deep learning

Abstract

The present invention provides a new energy ship fittings detection method and system based on deep learning, the method comprising the following steps: obtaining production drawings and multiple factory images of target ship fittings to be detected; generating a standard three-dimensional model of the target ship fittings based on the production drawings; marking the region of interest for each factory image based on the standard three-dimensional model; constructing a ship fittings segmentation model; completing the model training of the ship fittings segmentation model; collecting real-shot images of target ship fittings from multiple angles through multiple shooting positions; using the trained ship fittings segmentation model to segment the real-shot fittings images in all real-shot images, and generating a virtual three-dimensional model of the target ship fittings in combination with all the real-shot fittings images; comparing the model differences between the standard three-dimensional model and the virtual three-dimensional model, and generating the detection results of the target ship fittings based on the model differences. The present invention has the effect of higher detection accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of new energy ship detection, and specifically relates to a new energy ship accessories detection method and system based on deep learning. Background Art

[0002] With the continuous development of the new energy field, new energy technologies are gradually being applied to large ships. The application of new energy technologies in ships is of great significance. New energy ships use clean energy (such as electricity, hydrogen, wind, solar energy, etc.), which can significantly reduce the emission of carbon dioxide, sulfur oxides, nitrogen oxides and particulate matter, and reduce pollution to the atmosphere and marine environment. On the other hand, electric propulsion systems and other new energy technologies usually run more quietly, which helps to reduce the impact of ship noise on marine life and protect the marine ecosystem. Correspondingly, new energy ships will also add many more sophisticated and complex new energy accessories during the production and manufacturing process. For these sophisticated and complex new energy accessories, it is impossible to rely on human visual inspection methods for inspection, so machine vision inspection technology has become the development trend of new energy ship accessories inspection.

[0003] Machine vision technology is widely used, and there are many research results in the fields of face detection and recognition, mechanical parts quality inspection, optical character recognition, character coding detection, etc. The accessories of new energy ships often involve complex structures and new materials, which may have tiny, imperceptible defects. The reliability and stability of the functions of these important accessories will directly affect the safety of the application of new energy technologies on ships. In existing implementations, images of important new energy components are usually captured directly, and then the captured images are analyzed by machine vision technology to complete the defect detection of accessories. However, for lithium battery packs and generators with complex internal structures, or solar panels with strong reflective surfaces, the image analysis and processing process is prone to low accuracy due to the high image complexity or poor image clarity, which ultimately leads to missed defects. Summary of the invention

[0004] The present invention provides a new energy ship accessories detection method and system based on deep learning to solve the problem of low accuracy when image analysis is used to detect relatively complex new energy ship accessories.

[0005] In a first aspect, the present invention provides a new energy ship accessories detection method based on deep learning, the method comprising the following steps:

[0006] Acquire production drawings and multiple factory images of a target ship accessory to be inspected, wherein the multiple factory images include images of the target ship accessory taken based on different shooting angles;

[0007] Generating a standard three-dimensional model of the target ship accessory based on the production drawing;

[0008] Annotate a region of interest for each of the factory images based on the standard three-dimensional model;

[0009] Construct a ship parts segmentation model;

[0010] Inputting the factory image with the region of interest marked thereon into the ship parts segmentation model to complete the model training of the ship parts segmentation model;

[0011] Collecting real-shot images of the target ship accessories from multiple angles through multiple camera positions;

[0012] Using the trained ship accessory segmentation model to segment the real-shot accessory images from all the real-shot images, and combining all the real-shot accessory images to generate a virtual three-dimensional model of the target ship accessory;

[0013] The model difference between the standard three-dimensional model and the virtual three-dimensional model is compared, and a detection result of the target ship accessory is generated based on the model difference.

[0014] Optionally, the step of marking a region of interest for each of the factory images based on the standard three-dimensional model comprises the following steps:

[0015] pre-processing all said factory images;

[0016] Using an edge detection operator to identify and mark basic edges of the accessories in each of the factory images;

[0017] Fixing the image acquisition direction of the standard three-dimensional model;

[0018] Performing edge matching steps on each of the factory images respectively;

[0019] If the edge matching is successful, annotating the region of interest for the corresponding factory image according to the edge matching result;

[0020] If the edge matching fails, the region of interest is marked for the corresponding factory image using the level set method.

[0021] Optionally, the edge matching step includes the following steps:

[0022] Using the base edge of the accessory in the factory image as a comparison target;

[0023] Taking any two of the three-dimensional coordinate axes of the standard three-dimensional model as rotation axes, rotating the standard three-dimensional model based on the rotation axes and in accordance with a preset rotation direction and rotation step, and acquiring a model plane image of the standard three-dimensional model in the image acquisition direction after each rotation is completed;

[0024] After the model plane image is acquired each time, the edge detection operator is used to identify and mark the standard edges of the accessories in the model plane image;

[0025] Calculating edge similarity between the standard edge of the accessory and the basic edge of the accessory;

[0026] If the edge similarity exceeds a preset similarity threshold, the edge matching is determined to be successful, and the model plane image marked with the standard edge of the accessory is used as the edge matching result;

[0027] If the total rotation angle of any one of the rotation axes exceeds a preset angle threshold and the number of times the edge matching is determined to be successful is 0, it is determined that the edge matching fails.

[0028] Optionally, the step of marking the region of interest for the corresponding factory image according to the edge matching result comprises the following steps:

[0029] Adjusting the model plane image based on the basic edges of the accessories in the factory image so that all the basic edges of the accessories coincide with the standard edges of the accessories;

[0030] Generate edge feature points according to the standard edge of the accessory in the model plane image, and obtain feature point coordinates of the edge feature points;

[0031] generating a comparison feature point in the factory image according to the feature point coordinates;

[0032] Completing the basic edge of the accessory using the comparison feature points to obtain a completed edge of the accessory;

[0033] The area covered by the accessory completion edge in the factory image is marked as an area of ​​interest.

[0034] Optionally, the step of labeling the region of interest for the corresponding factory image by using the level set method comprises the following steps:

[0035] Constructing a level set function based on the factory image and initializing the level set function;

[0036] determining an energy functional corresponding to the level set function;

[0037] Using a variational method and a gradient descent algorithm to solve the extreme value of the energy functional, and iteratively updating the level set function;

[0038] The region of interest in the factory image is marked according to the iteratively updated level set function.

[0039] Optionally, the energy functional includes an internal energy functional and an external energy functional, and the formula of the energy functional is as follows:

[0040]

[0041] Where: represents the energy functional, represents the internal energy functional, represents the external energy functional, μ>0, and μ represents the internal energy coefficient;

[0042] The formula of the internal energy functional is as follows:

[0043]

[0044] Where: ∫∫ p represents the double integral, represents a gradient operator, u0(x, y) represents a point in the factory image;

[0045] The formula of the external energy functional is as follows:

[0046]

[0047] Where: λ represents the length weight coefficient, η represents the area weight coefficient, represents the length regularization term, represents the area regularization term, u0(x, y) represents a point in the factory image, δ(·) represents the Dirichlet function, H(·) represents the step function, and g(·) represents the stop speed function.

[0048] Optionally, the step of solving the extreme value of the energy functional by using a variational method and a gradient descent algorithm and iteratively updating the level set function comprises the following steps:

[0049] solving the minimum value of the energy functional by using a variational method to obtain an energy gradient of the energy functional;

[0050] Based on the energy gradient, the evolution equation of the level set function is derived through the gradient descent algorithm. The evolution equation formula is as follows:

[0051]

[0052] Where: represents the partial differential symbol, t represents the evolution time, and div(·) represents the divergence;

[0053] Using a regularized Dirichlet function to replace the Dirichlet function in the evolution equation;

[0054] The level set function is iteratively updated according to the evolution equation.

[0055] Optionally, the formula of the regularized Dirichlet function is as follows:

[0056]

[0057] Where: δ∈(s) represents the regularized Dirichlet function, ∈ represents the smoothing parameter.

[0058] Optionally, comparing the model difference between the standard three-dimensional model and the virtual three-dimensional model, and generating the detection result of the target ship accessory based on the model difference comprises the following steps:

[0059] Convert the standard three-dimensional model and the virtual three-dimensional model into model point clouds to obtain standard three-dimensional point cloud data of the standard three-dimensional model and virtual three-dimensional point cloud data of the virtual three-dimensional model;

[0060] Performing data preprocessing on the standard three-dimensional point cloud data and the virtual three-dimensional point cloud data;

[0061] Performing point cloud registration on the standard three-dimensional point cloud data and the virtual three-dimensional point cloud data by using a point cloud registration algorithm, so that the standard three-dimensional point cloud data and the virtual three-dimensional point cloud data achieve point cloud alignment;

[0062] Calculating the point cloud distances in the standard three-dimensional point cloud data and the virtual three-dimensional point cloud data respectively by using a nearest neighbor search algorithm;

[0063] If there is a distance difference between the point cloud distance of any target point cloud data in the virtual three-dimensional point cloud data and the point cloud distance of the corresponding point cloud data in the standard three-dimensional point cloud data, the detection result of the target ship accessory is generated by combining the distance difference and the point cloud position of the target point cloud data in the virtual three-dimensional point cloud data.

[0064] In a second aspect, the present invention also provides a new energy ship accessories detection system based on deep learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the computer program.

[0065] The beneficial effects of the present invention are:

[0066] By using automated image segmentation and 3D modeling technology, the time and labor for manual inspection and measurement of new energy ship accessories can be greatly reduced. In addition, once trained, the model can quickly process new images, improving production efficiency. Machine learning models and 3D modeling technology can provide higher accuracy and consistency than traditional methods. By comparing standard 3D models and virtual 3D models, the differences in new energy ship accessories can be accurately identified and quantified.

[0067] Through automatic detection and comparison, problems in the production process can be discovered earlier, thereby providing timely feedback, improving the production process, and improving the overall quality of new energy ship accessories. The automated inspection process can reduce errors and rework, thereby saving production costs. In addition, by detecting problems early, greater repair costs and potential quality issues can be avoided. Once the model is built and trained, it can be easily applied to other similar new energy ship accessories, or adjusted and optimized when necessary. This flexibility and scalability makes this approach very suitable for large-scale production environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 This is a flow chart of a new energy ship accessories detection method based on deep learning in one embodiment of the present application.

[0069] Figure 2 This is a schematic diagram of a process for labeling regions of interest for each factory image based on a standard three-dimensional model in one implementation of the present application.

[0070] Figure 3 This is a schematic diagram of the process of marking a region of interest for a corresponding factory image according to edge matching results in one embodiment of the present application.

[0071] Figure 4 This is a schematic diagram of a process of labeling a region of interest for a corresponding factory image using a level set method in one embodiment of the present application. DETAILED DESCRIPTION

[0072] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0073] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0074] Figure 1 FIG. 1 is a flow chart of a method for detecting new energy ship parts based on deep learning in one embodiment. It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps in the above method may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps. Figure 1 As shown, a new energy ship accessories detection method based on deep learning disclosed in the present invention specifically includes the following steps:

[0075] S101. Obtain production drawings and multiple factory images of the target ship parts to be inspected.

[0076] In this step, it is necessary to obtain the production drawings of the target ship accessories, which refers to the design drawings or CAD files of the accessories, including the size, shape and other information of the accessories. At the same time, it is also necessary to collect multiple factory images, which are actual images taken during the production process of the accessories for subsequent analysis and comparison. Among them, multiple factory images include images of the target ship accessories taken from different shooting angles.

[0077] S102. Generate a standard three-dimensional model of the target ship accessories based on the production drawings.

[0078] In this step, the detailed design drawings of the target ship accessories are obtained. These drawings usually contain key design information such as size, shape, tolerance, etc. At the same time, the factory images of the target ship accessories are collected. These images reflect the appearance of the actual produced accessories, including shots from multiple angles, so as to fully capture the characteristics of the accessories. Using the obtained production drawings, a standard 3D model of the target ship accessories can be generated through computer-aided design software or 3D modeling tools. This step converts the design drawings of the accessories into a specific 3D model for subsequent comparison and analysis.

[0079] S103. Annotate the region of interest for each factory image based on the standard three-dimensional model.

[0080] In this step, the standard 3D model is used to annotate the region of interest for each factory image. The region of interest refers to the location and boundaries of the accessories in the image. Based on the 3D model, the corresponding target ship accessory area in the factory image is annotated. This can be done using image processing software or by using automated algorithms to identify the edges and features of the accessories for annotation. These annotations define the "ground truth" in subsequent model training.

[0081] S104. Construct a ship parts segmentation model.

[0082] In this step, select a suitable machine learning or deep learning framework, such as TensorFlow or PyTorch, to build a neural network model for image segmentation. The network architecture may be U-Net, Mask R-CNN, etc., which can identify and segment new energy ship accessories in the image. The model is trained to learn the features and boundaries of the accessories.

[0083] S105. Input the factory image with the region of interest marked into the ship parts segmentation model to complete the model training of the ship parts segmentation model.

[0084] In this step, the segmentation model is trained through supervised learning using labeled factory images as a training dataset. During the training process, the model adjusts its weights and biases to improve accuracy by minimizing the difference between the prediction and the true value (for example, using the cross entropy loss function).

[0085] S106. Collect real-shot images of target ship accessories from multiple angles through multiple camera positions.

[0086] In this step, multiple cameras are set up to shoot the target ship accessories from different angles to obtain a series of real-shot images. These images should cover all visible surfaces of the accessories to facilitate subsequent 3D reconstruction.

[0087] S107. Using the trained ship accessory segmentation model, segment the real-shot accessory images from all the real-shot images, and generate a virtual three-dimensional model of the target ship accessory by combining all the real-shot accessory images.

[0088] In this step, the trained segmentation model is used to process the real-life image to extract the precise contour of the accessory. Then, a 3D reconstruction technique, such as stereovision or structured light scanning, is used to combine the extracted contour information to generate a virtual 3D model of the target accessory.

[0089] S108. Compare the model differences between the standard three-dimensional model and the virtual three-dimensional model, and generate detection results of the target ship accessories based on the model differences.

[0090] In this step, the virtual 3D model is compared with the standard 3D model, which may involve geometric analysis, difference detection algorithms, or even machine learning methods to evaluate the differences between the two models. The differences can be quantified based on aspects such as geometry, size, surface quality, etc. Based on the preset tolerance range, it is determined whether the accessory meets the quality standards and an inspection report is output.

[0091] In one implementation manner, step S103 specifically includes the following steps:

[0092] S201. Preprocess all factory images;

[0093] S202. Using edge detection operators to identify and mark the basic edges of the accessories in each factory image;

[0094] S203. Fixing the image acquisition direction of the standard three-dimensional model;

[0095] S204. Perform edge matching steps on each factory image respectively;

[0096] S205. If the edge matching is successful, annotate the region of interest for the corresponding factory image according to the edge matching result;

[0097] S206. If the edge matching fails, the region of interest is marked for the corresponding factory image using the level set method.

[0098] In this embodiment, the preprocessing step is intended to improve the image data so that subsequent processing steps are more efficient and accurate. Preprocessing may include operations such as denoising, adjusting contrast, normalization, filtering, etc. to eliminate noise and inconsistencies in the image and enhance image features. Edge detection operators (such as Sobel, Canny, etc.) are used to identify obvious boundaries in the image. These operators detect edges by calculating the gradient of image brightness. The identified edges can help locate the outline of new energy ship accessories in the image.

[0099] In order to ensure consistency between factory images and standard 3D models, the image acquisition direction is standardized. This means that all images should be taken from the same or predefined angle to facilitate subsequent edge matching and 3D model comparison. Edge matching involves comparing the edges detected in the factory images with the expected edges of the standard 3D model. This can be achieved through feature matching algorithms such as template matching, feature point matching (such as SIFT, SURF), etc. Successful edge matching means that the position and shape of the accessories in the image have been determined. Based on this information, the region of interest can be directly annotated on the image to provide accurate data for subsequent segmentation model training.

[0100] If edge matching fails, the level set method is used to annotate the region of interest for the corresponding factory image. The level set method is an image segmentation technique based on a mathematical model that uses contour evolution to detect the edges of an object. When traditional edge matching methods fail, the level set method can be used as a more robust alternative. It initializes a contour (usually a simple geometric shape such as a circle) and then lets the contour evolve in the image until it fits the edge of the target object. This method is resistant to noise and other interference in the image, so it is suitable for objects with unclear edges or partially occluded.

[0101] In this embodiment, the edge matching step specifically includes the following steps:

[0102] The base edge of the parts in the factory image is used as the comparison target;

[0103] Taking any two coordinate axes of the three-dimensional coordinate axes of the standard three-dimensional model as rotation axes, rotating the standard three-dimensional model based on the rotation axes and according to a preset rotation direction and rotation step, and acquiring a model plane image of the standard three-dimensional model in an image acquisition direction after each rotation is completed;

[0104] After each model plane image is acquired, the edge detection operator is used to identify and mark the standard edges of the accessories in the model plane image;

[0105] Calculate the edge similarity between the standard edge of the accessory and the basic edge of the accessory;

[0106] If the edge similarity exceeds the preset similarity threshold, the edge matching is determined to be successful, and the model plane image marked with the standard edge of the accessory is used as the edge matching result;

[0107] If the total rotation angle of any rotation axis exceeds the preset angle threshold and the number of times the edge matching is successful is 0, the edge matching is determined to have failed.

[0108] In this embodiment, in image processing, an edge is a distinct boundary between an object and a background. By using the basic edge of the parts in the factory image as a comparison target, the target object in the image can be more accurately located and identified. Based on the principles of three-dimensional computer graphics. Rotating the model allows the model to be observed from different angles, which helps to more fully understand the shape of the model. By fixing the image acquisition direction, it can be ensured that the acquired model plane image has a consistent perspective with the factory image, thereby facilitating comparison.

[0109] The standard three-dimensional model includes three coordinate axes, namely, the x-axis, the y-axis and the z-axis. Any two of the coordinate axes are used as rotation axes. The standard three-dimensional model is rotated based on the rotation axes and in accordance with the preset rotation direction and rotation step length. For example: Assuming that the x-axis and the y-axis are used as rotation axes, the preset rotation direction is clockwise rotation, and the rotation step length is 1°, the standard three-dimensional model is first rotated 1° clockwise along the x-axis, and then the first model plane image of the standard three-dimensional model is obtained by screenshot in the image acquisition direction; the standard three-dimensional model is then rotated 1° clockwise along the y-axis, and then the second model plane image of the standard three-dimensional model is obtained by screenshot in the image acquisition direction; then the standard three-dimensional model is continuously rotated clockwise along the y-axis until the rotation angle of the y-axis reaches 359°; at this time, the standard three-dimensional model is rotated 1° clockwise along the x-axis, and then the standard three-dimensional model is continuously rotated clockwise along the y-axis until the rotation angle of the y-axis reaches 359° again; this cycle is repeated until the edge matching is determined to be successful, or the number of successful edge matching is determined to be 0 and the total rotation angle of the x-axis exceeds the preset angle threshold.

[0110] Each time the model plane image is acquired, the edge detection operator is used to identify and mark the accessory standard edge of the model plane image. The edge detection operator can identify the boundaries in the image, thereby helping us locate and identify the target object in the image. Edge similarity is usually measured by calculating the distance or angle difference between two sets of edge points. The closer the two sets of edge points are, the higher their similarity is. If the edge similarity exceeds the preset similarity threshold, the edge matching is determined to be successful, and the model plane image marked with the accessory standard edge is used as the edge matching result. The similarity threshold is a preset standard used to determine whether the two sets of edges are close enough. If the edge similarity exceeds this threshold, the edge matching is considered to be successful. If the total rotation angle of any rotation axis exceeds the preset angle threshold and the number of successful edge matching is determined to be 0, the edge matching is considered to have failed. The purpose of this step is to prevent unlimited rotation and matching. If no matching edge is found after rotating to a certain angle, the edge matching is considered to have failed.

[0111] In one implementation manner, the step of marking the region of interest for the corresponding factory image according to the edge matching result specifically includes the following steps:

[0112] S301. Adjust the model plane image based on the basic edges of the accessories in the factory image so that all basic edges of the accessories coincide with the standard edges of the accessories;

[0113] S302. Generate edge feature points according to the standard edges of the accessories in the model plane image, and obtain the feature point coordinates of the edge feature points;

[0114] S303. Generate control feature points in the factory image according to the feature point coordinates;

[0115] S304. Use the reference feature points to complete the basic edge of the accessory to obtain the completed edge of the accessory;

[0116] S305: Mark the area covered by the edge of the accessory completion in the factory image as an area of ​​interest.

[0117] In this embodiment, the standard edge of the accessory in the model plane image is aligned with the basic edge of the accessory in the factory image through image registration technology. This usually involves geometric transformations such as rotation, scaling and translation. The goal of registration is to maximize the coverage and alignment between corresponding features in the two images, so as to achieve edge overlap. Edge feature points are key positions of the edge in the image and can be used to describe the shape and structure of the edge. These points can be extracted by detecting corner points, curvature change points or other significant features on the edge. Each feature point has its coordinate position in the image. Edge feature points are generated according to the standard edge of the accessory in the model plane image, and the feature point coordinates of the edge feature points are obtained.

[0118] Using the transformation parameters obtained after image registration, the coordinates of the feature points in the model plane image can be mapped to the factory image to generate the corresponding comparison feature points. This requires ensuring the accuracy of the registration step so that the feature points can correspond correctly. In the factory image, the edges detected and marked by the edge detection operator may be incomplete. Using the comparison feature points, the edges can be completed by interpolation, morphological operations or other edge connection techniques to restore the complete accessory edges. This usually requires considering the continuity and smoothness of the edges. Once the completed edge of the accessory is obtained, the area within the edge can be segmented out by image segmentation techniques such as threshold segmentation, region growing or contour tracking. This area is the area of ​​interest, that is, the part containing the accessory. Marking this area is conducive to subsequent image analysis, such as feature extraction, target recognition or quality inspection.

[0119] In one implementation manner, the step of marking the region of interest for the corresponding factory image by using the level set method specifically includes the following steps:

[0120] S401. Constructing a level set function based on the factory image and initializing the level set function;

[0121] S402. Determine the energy functional corresponding to the level set function;

[0122] S403. Using the variational method and gradient descent algorithm to solve the extreme value of the energy functional, and iteratively update the level set function;

[0123] S404. Mark the region of interest in the factory image according to the iteratively updated level set function.

[0124] In this embodiment, a level set function is constructed based on the factory image and initialized: the level set function is usually expressed as a high-dimensional function The value of each point represents the distance from the point to the interface. The interface itself is represented by the level set function Represents. Initializing the level set function usually involves choosing an initial interface and then computing the corresponding level set function. For example, if the region of interest is an object in the center of the image, the initial interface can be a circle centered at the object and slightly larger than the object size.

[0125] The energy functional is a function defined on the level set function that measures the "energy" of the current interface. In the context of image segmentation, this energy is usually designed as the consistency of the interface with the image data (for example, the difference in pixel intensity inside and outside the interface) and the smoothness of the interface. The goal is to find the interface shape that minimizes this energy functional. The calculus of variations is a method for finding a function that minimizes the energy functional. By calculating the variational derivative of the energy functional with respect to the level set function, a partial differential equation (PDE) can be obtained. The gradient descent algorithm is used to solve this PDE, by iteratively updating the level set function to gradually reduce the value of the energy functional. At each iteration, the level set function is adjusted according to its gradient descent direction, so that the interface evolves in the direction of reducing the energy functional.

[0126] As the level set function is iterated, the interface gradually approaches the real boundary in the image. When the iteration ends or a certain termination condition is reached (such as the energy is reduced to below a certain threshold), the level set function The contour lines of the segmented image can be used to find the segmented interface. This interface is the boundary of the region of interest in the image. Then, the region of interest can be marked in the image based on this boundary. A major advantage of the level set method in image segmentation is that it can naturally handle topological changes in the interface (such as splitting and merging), making it very suitable for the recognition and tracking of complex or irregularly shaped objects.

[0127] In this embodiment, the energy functional includes an internal energy functional and an external energy functional. The formula of the energy functional is as follows:

[0128]

[0129] Where: represents the energy functional, represents the internal energy functional, represents the external energy functional, μ>0, μ represents the internal energy coefficient;

[0130] The formula for the internal energy functional is as follows:

[0131]

[0132] Where: ∫∫ D represents the double integral, represents the gradient operator, u0(x, y) represents a point in the factory image;

[0133] The formula for the external energy functional is as follows:

[0134]

[0135] Where: λ represents the length weight coefficient, η represents the area weight coefficient, represents the length regularization term, represents the area regularization term, u0(x, y) represents the point in the factory image, δ(·) represents the Dirichlet function, H(·) represents the step function, and g(·) represents the stopping speed function.

[0136] In one embodiment, the step of solving the extreme value of the energy functional by using the variational method and the gradient descent algorithm and iteratively updating the level set function specifically includes the following steps:

[0137] The variational method is used to solve the minimum value of the energy functional and obtain the energy gradient of the energy functional.

[0138] Based on the energy gradient and through the gradient descent algorithm, the evolution equation of the level set function is derived. The evolution equation formula is as follows:

[0139]

[0140] Where: represents the partial differential symbol, t represents the evolution time, and div(·) represents the divergence;

[0141] Use the regularized Dirichlet function to replace the Dirichlet function in the evolution equation;

[0142] The level set function is iteratively updated according to the evolution equation.

[0143] In this embodiment, the minimum value of the energy functional is solved by the variational method to obtain the energy gradient of the energy functional. The variational method is a method for solving the extreme value of the energy functional, in which the variational derivative of the energy functional with respect to the level set function is calculated. This variational derivative can be regarded as the "energy gradient" of the energy functional. This gradient indicates the direction and magnitude of the change of the energy functional with the level set function, which can be used to guide the update direction of the level set function.

[0144] The evolution equation of the level set function is derived based on the energy gradient and through the gradient descent algorithm. The gradient descent algorithm is an optimization algorithm that gradually reduces the value of the objective function by updating the function along the negative direction of the gradient. In the level set method, the evolution equation of the level set function can be obtained by multiplying the energy gradient with a time step and subtracting this value from the current level set function. This equation describes the change of the level set function over time, that is, how the interface evolves over time.

[0145] The Dirichlet energy is a form of regularization term that is used to maintain the smoothness of the interface. In practical applications, the direct use of the Dirichlet energy may lead to numerical problems such as instability and oscillation. Therefore, a regularized version of the Dirichlet energy is often used, such as by adding a small positive value (called the regularization parameter) to the Dirichlet energy to avoid division by zero and improve the stability of numerical calculations. Once the evolution equation is derived, the level set function can be updated in an iterative manner. In each iteration, a new level set function is calculated according to the evolution equation and the old function is replaced by this new function. This process is repeated until a certain stopping condition is reached, such as the change in the interface evolution is lower than a certain threshold, or the evolution reaches a predetermined number of iterations.

[0146] The above steps together form the core of the level set method, which uses numerical methods to realize the evolution of interfaces in the image, thereby achieving image segmentation. This method is particularly suitable for capturing complex or dynamically changing interfaces and can handle topological changes such as the merging and splitting of objects. In practical applications, these steps usually require the help of numerical analysis and computer vision expertise to implement.

[0147] In this embodiment, the formula of the regularized Dirichlet function is as follows:

[0148]

[0149] Where: ε (s) represents the regularized Dirichlet function, ∈ represents the smoothing parameter.

[0150] In one implementation manner, step S108 specifically includes the following steps:

[0151] Convert the standard 3D model and the virtual 3D model into point cloud data to obtain standard 3D point cloud data of the standard 3D model and virtual 3D point cloud data of the virtual 3D model;

[0152] Perform data preprocessing on standard 3D point cloud data and virtual 3D point cloud data;

[0153] The standard 3D point cloud data and the virtual 3D point cloud data are registered by a point cloud registration algorithm, so that the standard 3D point cloud data and the virtual 3D point cloud data can be aligned.

[0154] The nearest neighbor search algorithm is used to calculate the point cloud distances in the standard 3D point cloud data and the virtual 3D point cloud data respectively;

[0155] If there is a distance difference between the point cloud distance of any target point cloud data in the virtual three-dimensional point cloud data and the point cloud distance of the corresponding point cloud data in the standard three-dimensional point cloud data, the detection result of the target ship accessories is generated by combining the distance difference and the point cloud position of the target point cloud data in the virtual three-dimensional point cloud data.

[0156] In this embodiment, the standard 3D model and the virtual 3D model are converted into point clouds to obtain standard 3D point cloud data of the standard 3D model and virtual 3D point cloud data of the virtual 3D model. Point cloud is a data type used to represent a 3D object or scene, consisting of a set of points scattered in a 3D space. Point cloud conversion is the process of converting a 3D model into a point cloud, usually by sampling the surface of the model. In this step, both the standard model and the virtual model are converted into point clouds for subsequent processing and comparison.

[0157] Data preprocessing is performed on the standard 3D point cloud data and the virtual 3D point cloud data. The preprocessing steps usually include denoising, filtering, normalization, etc. to improve data quality and reduce the complexity of subsequent processing. For example, noise points can be removed by various filters, or the point cloud data can be normalized to a unified scale and coordinate system by transformation. The standard 3D point cloud data and the virtual 3D point cloud data are point cloud registered by the point cloud registration algorithm, so that the standard 3D point cloud data and the virtual 3D point cloud data are point cloud aligned. Point cloud registration is the process of finding the best correspondence between two point clouds so that they are aligned in space. Commonly used registration algorithms include the iterative closest point (ICP) algorithm, the basic idea of ​​which is to iteratively find the nearest point pairs and find the optimal alignment transformation by minimizing the distance between these point pairs.

[0158] The nearest neighbor search algorithm is used to calculate the distance between the point clouds in the standard 3D point cloud data and the virtual 3D point cloud data. The nearest neighbor search is an algorithm in computer science that is used to find the nearest neighbor of a given point in a point set. In this context, the nearest neighbor search is used to calculate the distance from each point to its nearest neighbor, and these distances can be used to compare the difference between the two point clouds. If there is a distance difference between the point cloud distance of any target point cloud data in the virtual 3D point cloud data and the point cloud distance of the corresponding point cloud data in the standard 3D point cloud data, the detection result of the target ship accessories is generated by combining the distance difference and the point cloud position of the target point cloud data in the virtual 3D point cloud data. If the position of a point in the virtual model is significantly different from its position in the standard model, then the point may represent an error or anomaly. This information can be used to detect and locate problems with new energy ship accessories, and finally the detection result of the target ship accessories is generated by combining the distance difference and the point cloud position of the target point cloud data in the virtual 3D point cloud data.

[0159] In one implementation manner, step S108 specifically includes the following steps:

[0160] Obtain voxelized data of the solid 3D model and the standard 3D model, and divide the 3D space into small cubes. Specifically, a voxelization algorithm such as Octree or Grid-based method can be used.

[0161] Preprocess the voxels of solid 3D models and standard 3D models, such as removing holes and smoothing.

[0162] Compare the differences between two voxel models. You can calculate the difference in the number of voxels, the difference in position, or use shape descriptors (such as shape histograms, normal histograms) to compare the shape differences between the two models.

[0163] Based on the comparison results, a visual display can be made or a difference report can be generated to analyze and evaluate the differences between the solid 3D model and the standard 3D model, and finally the detection results of the target ship accessories can be generated based on the difference results.

[0164] In this embodiment, obtaining voxelized data of the solid three-dimensional model and the standard three-dimensional model is the first step in making a high-precision comparison. Specifically, a voxelization algorithm, such as an Octree or Grid-based method, can be used. The Octree algorithm recursively divides the three-dimensional space into eight subspaces, which can effectively manage and process complex three-dimensional data, while the Grid-based method divides the three-dimensional space evenly into small cubes of fixed size. This method is intuitive and easy to implement. Next, preprocessing the voxels of the solid three-dimensional model and the standard three-dimensional model is a key step to ensure the accuracy of the comparison. Preprocessing includes removing holes and smoothing. Removing holes can be done by filling in undefined areas with a filling algorithm, and smoothing can use a filtering algorithm, such as a Gaussian filter, to reduce noise and detail errors, thereby obtaining a more accurate model.

[0165] Comparing the difference between two voxel models is the core step. The quantitative difference and positional difference of voxels can be calculated. The quantitative difference is measured by counting the number of voxels, and the positional difference can be achieved by calculating the coordinate difference of the voxels. In addition, shape descriptors can be used to compare the shape differences of the two models, such as shape histograms and normal histograms. The shape histogram counts the distribution of the surface shape of the model, and the normal histogram counts the distribution of the normal direction of the surface of the model. These descriptors can provide more detailed shape comparison information.

[0166] Based on the comparison results, a visual display or a difference report can be generated. The visual display can intuitively display the difference areas of the two models by color coding, such as using red to mark areas with larger differences and green to mark areas with smaller differences. The difference report can list in detail the comparison results of the voxel quantity difference, position difference, and shape descriptor, thereby providing data support for analyzing and evaluating the differences between the solid 3D model and the standard 3D model. Finally, the detection results of the target ship accessories are generated in combination with the difference results. The qualification of the accessories can be determined by setting the difference threshold, further improving the automation and accuracy of the detection. Through this systematic process, the quality of ship accessories can be efficiently detected and evaluated to ensure that they meet the design standards and usage requirements.

[0167] The present invention also discloses a new energy ship accessories detection system based on deep learning, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the new energy ship accessories detection method based on deep learning as described in any one of the above embodiments is implemented.

[0168] Among them, the processor can adopt a central processing unit (CPU). Of course, according to actual usage, other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. can also be adopted. The general-purpose processor can adopt a microprocessor or any conventional processor, etc., and this application does not impose any restrictions on this.

[0169] Among them, the memory can be an internal storage unit of a computer device, such as a hard disk or memory of a computer device, or an external storage device of a computer device, such as a plug-in hard disk, a smart memory card (SMC), a secure digital card (SD) or a flash memory card (FC) equipped on the computer device, etc., and the memory can also be a combination of an internal storage unit and an external storage device of a computer device. The memory is used to store computer programs and other programs and data required by the computer device. The memory can also be used to temporarily store data that has been output or is to be output, and this application does not impose any restrictions on this.

[0170] A person skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of protection of the present application is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present application as above, which are not provided in detail for the sake of simplicity.

[0171] One or more embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the present application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application should be included in the protection scope of the present application.

Claims

1. A new energy ship accessories detection method based on deep learning, characterized in that: The steps include: Acquire production drawings and multiple factory images of a target ship component to be inspected, wherein the multiple factory images include images of the target ship component taken based on different shooting angles; Generating a standard three-dimensional model of the target ship accessory based on the production drawing; pre-processing all said factory images; Using an edge detection operator to identify and mark basic edges of the accessories in each of the factory images; Fixing the image acquisition direction of the standard three-dimensional model; Performing edge matching steps on each of the factory images respectively; The edge matching step comprises the following steps: Using the base edge of the accessory in the factory image as a comparison target; Taking any two of the three-dimensional coordinate axes of the standard three-dimensional model as rotation axes, rotating the standard three-dimensional model based on the rotation axes and in accordance with a preset rotation direction and rotation step, and acquiring a model plane image of the standard three-dimensional model in the image acquisition direction after each rotation is completed; After the model plane image is acquired each time, the edge detection operator is used to identify and mark the standard edges of the accessories in the model plane image; Calculating edge similarity between the standard edge of the accessory and the basic edge of the accessory; If the edge similarity exceeds a preset similarity threshold, the edge matching is determined to be successful, and the model plane image marked with the standard edge of the accessory is used as the edge matching result; If the total rotation angle of any of the rotation axes exceeds the preset angle threshold and the number of times the edge matching is successful is 0, the edge matching is determined to have failed; If the edge matching is successful, annotating the region of interest for the corresponding factory image according to the edge matching result; If the edge matching fails, annotating the region of interest for the corresponding factory image by using the level set method; Construct a ship parts segmentation model; Inputting the factory image with the region of interest marked thereon into the ship parts segmentation model to complete the model training of the ship parts segmentation model; Collecting real-shot images of the target ship accessories from multiple angles through multiple camera positions; Using the trained ship accessory segmentation model to segment the real-shot accessory images from all the real-shot images, and combining all the real-shot accessory images to generate a virtual three-dimensional model of the target ship accessory; The model difference between the standard three-dimensional model and the virtual three-dimensional model is compared, and a detection result of the target ship accessory is generated based on the model difference.

2. The deep learning-based new energy ship accessories detection method according to claim 1 is characterized in that: The step of marking the region of interest for the corresponding factory image according to the edge matching result comprises the following steps: Adjusting the model plane image based on the basic edges of the accessories in the factory image so that all the basic edges of the accessories coincide with the standard edges of the accessories; Generate edge feature points according to the standard edge of the accessory in the model plane image, and obtain feature point coordinates of the edge feature points; generating a comparison feature point in the factory image according to the feature point coordinates; Completing the basic edge of the accessory using the comparison feature points to obtain a completed edge of the accessory; The area covered by the accessory completion edge in the factory image is marked as an area of ​​interest.

3. The deep learning-based new energy ship accessories detection method according to claim 1 is characterized in that: The step of marking the region of interest for the corresponding factory image by the level set method comprises the following steps: Constructing a level set function based on the factory image and initializing the level set function; determining an energy functional corresponding to the level set function; Using a variational method and a gradient descent algorithm to solve the extreme value of the energy functional, and iteratively updating the level set function; The region of interest in the factory image is marked according to the iteratively updated level set function.

4. The deep learning-based new energy ship accessories detection method according to claim 3 is characterized in that: The energy functional includes an internal energy functional and an external energy functional. The formula of the energy functional is as follows: ， Where: represents the energy functional, represents the internal energy functional, represents the external energy functional, , represents the internal energy coefficient; The formula of the internal energy functional is as follows: ， Where: represents the double integral, represents the gradient operator, represents a point in the factory image; The formula of the external energy functional is as follows: ， Where: represents the length weight coefficient, represents the area weight coefficient, represents the length regularization term, represents the area regularization term, represents a point in the factory image, represents the Dirichlet function, represents a step function, Represents the stop speed function.

5. The deep learning-based new energy ship accessories detection method according to claim 4 is characterized in that: The method of solving the extreme value of the energy functional by using the variational method and the gradient descent algorithm and iteratively updating the level set function comprises the following steps: solving the minimum value of the energy functional by using a variational method to obtain an energy gradient of the energy functional; Based on the energy gradient, the evolution equation of the level set function is derived through the gradient descent algorithm. The evolution equation formula is as follows: ， Where: represents the partial differential symbol, represents the evolution time, is the divergence representation; Using a regularized Dirichlet function to replace the Dirichlet function in the evolution equation; The level set function is iteratively updated according to the evolution equation.

6. The deep learning-based new energy ship accessories detection method according to claim 5 is characterized in that: The formula of the regularized Dirichlet function is as follows: ， Where: represents the regularized Dirichlet function, represents the smoothing parameter.

7. The deep learning-based new energy ship accessories detection method according to claim 1 is characterized in that: The comparing the model difference between the standard three-dimensional model and the virtual three-dimensional model and generating the detection result of the target ship accessories based on the model difference comprises the following steps: Convert the standard three-dimensional model and the virtual three-dimensional model into model point clouds to obtain standard three-dimensional point cloud data of the standard three-dimensional model and virtual three-dimensional point cloud data of the virtual three-dimensional model; Performing data preprocessing on the standard three-dimensional point cloud data and the virtual three-dimensional point cloud data; Performing point cloud registration on the standard three-dimensional point cloud data and the virtual three-dimensional point cloud data by using a point cloud registration algorithm, so that the standard three-dimensional point cloud data and the virtual three-dimensional point cloud data achieve point cloud alignment; Calculating the point cloud distances in the standard three-dimensional point cloud data and the virtual three-dimensional point cloud data respectively by using a nearest neighbor search algorithm; If there is a distance difference between the point cloud distance of any target point cloud data in the virtual three-dimensional point cloud data and the point cloud distance of the corresponding point cloud data in the standard three-dimensional point cloud data, the detection result of the target ship accessory is generated by combining the distance difference and the point cloud position of the target point cloud data in the virtual three-dimensional point cloud data.

8. A new energy ship parts detection system based on deep learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

Images