A method for positioning the edge of a port hatch cover based on Unet semantic segmentation

Abstract

The application relates to the technical field of image processing, and discloses a port hatch cover plate edge positioning method based on Unet semantic segmentation. The port hatch cover plate edge positioning method based on Unet semantic segmentation trains a Unet model by using processed image data, so that the model can accurately identify the edge position of the hatch cover plate. The trained Unet model is tested and optimized to improve the accuracy and generalization ability of the model, thereby determining a set of edge points of the hatch cover plate. A contour extraction method is used to extract the edge of the hatch cover plate, convex hull detection is performed, and a convex hull is drawn. The set of corner points is traversed, the minimum value of the Euclidean distance from the edge corner point to the hatch cover plate normal rectangular corner point is calculated, thereby determining the optimal corner point of the edge of the hatch cover plate, and the edge positioning work is completed. The method has higher precision and efficiency, is lower in cost, and is suitable for popularization and use.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a method for locating the edge of a port hatch cover based on Unet semantic segmentation. Background Technology

[0002] With the economic development of countries and regions, the demand for resources is constantly increasing. Ports, as important nodes in logistics and transportation, can improve the efficiency of import and export trade and reduce costs. Therefore, port transportation construction has become an important means to promote economic development, and hatch cover edge positioning technology has received widespread attention and application in recent years. The position of hatch covers is a key technical parameter that needs to be accurately measured and monitored in real time during transportation, playing a crucial role in achieving accurate hatch cover loading, unloading, and transportation, and improving port operational efficiency and safety. With the rapid development of deep learning and computer vision technologies in recent years, image processing-based port hatch cover edge positioning methods have gradually gained attention. Among them, the Unet semantic segmentation-based method has advantages such as high accuracy, high efficiency, and low cost.

[0003] Traditional methods for locating the edges of port hatch covers typically employ visual or mechanical positioning, which suffer from low accuracy, low efficiency, or high cost, thus affecting practical usability. Therefore, we propose a port hatch cover edge positioning method based on Unet semantic segmentation. Summary of the Invention

[0004] The purpose of this invention is to provide a method for locating the edge of a port hatch cover based on Unet semantic segmentation, so as to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a method for locating the edge of a port hatch cover based on Unet semantic segmentation, the method comprising the following steps:

[0006] S1. Data Acquisition: First, it is necessary to collect image data of the port hatch cover, which can be obtained by taking images or videos with a camera.

[0007] S2. Data preprocessing: Preprocess the acquired image data, including noise removal, contrast enhancement, data augmentation, etc., to improve image quality;

[0008] S3. Training the Unet model: Use the processed image data to train the Unet model so that the model can accurately identify the edge position of the hatch cover.

[0009] S4. Model Testing and Optimization: Test and optimize the trained Unet model to improve its accuracy and generalization ability.

[0010] S5. Semantic segmentation: The trained Unet model is used to perform semantic segmentation on the new port hatch cover image to determine the set of edge points of the hatch cover.

[0011] S6. Edge localization: The edge of the hatch cover is extracted using the contour extraction method, the convex hull is detected and drawn, the set of corner points is traversed, and the minimum Euclidean distance from the edge corner point to the corner point of the rectangular hatch cover is calculated to determine the optimal corner point of the hatch cover edge and complete the edge localization work.

[0012] Optionally, S1 further includes:

[0013] S11. Search for and retrieve the camera information list, then open the corresponding camera according to the camera information list and retrieve the camera object;

[0014] S12. Select either internal trigger acquisition mode or external trigger acquisition mode. Internal trigger acquisition includes continuous acquisition and single-frame acquisition, while external trigger acquisition includes soft trigger and hard trigger.

[0015] S13. Open the acquisition stream;

[0016] S14. If the internal trigger acquisition mode is used, the image can be acquired by opening the acquisition stream; if the external trigger acquisition mode is used, a trigger command is sent.

[0017] S15. Capture stream and capture images. There are two ways to capture stream and capture images: active capture stream and capture stream and capture stream and capture. If you use the callback capture stream and capture method, the callback function registration operation should be placed before the operation of opening the capture stream.

[0018] S16. Close the acquisition stream;

[0019] S17. Turn off the camera.

[0020] Optionally, S2 further includes:

[0021] S21. Data cleaning: remove outliers, handle missing values, remove redundant data, screen the original images, remove data with poor lighting conditions, blurry images, or excessive lighting conditions, and select some clear images that contain hatch covers for backup.

[0022] S22. Perform data enhancement processing.

[0023] Optionally, the enhancement process includes:

[0024] Cropping and scaling processes change the size and perspective of an image by adding background around it or removing some edge pixels, and by scaling the image up or down.

[0025] Rotation and flipping processing changes the viewpoint and orientation of an image by rotating it by a certain angle, and changes its symmetry and orientation by flipping it horizontally or vertically by a certain angle.

[0026] Color transformation processing increases the diversity and quantity of data by changing the brightness, contrast, and saturation of an image;

[0027] Grayscale conversion simplifies image processing by converting color images to grayscale.

[0028] Optionally, S3 further includes:

[0029] S31. Data annotation and processing: Select 1000 prepared images including hatch covers, with a size of 512×512. Use Labelme annotation software to create a dataset, customize label names, mark the hatch covers in the image data, save the annotation results, generate XML label data, and convert it into the corresponding Mask image data.

[0030] S32. Constructing a model for semantic segmentation of hatch covers. The network structure of the model includes an encoder and a decoder. The encoder is responsible for extracting the feature information of the image, and the decoder restores the feature information to the same size as the original image by upsampling. The encoder and decoder transmit information through skip connections.

[0031] S33. Divide the dataset into training and validation sets according to a set ratio of 9:1 for training and validating the model.

[0032] S34. Configure parameter processing: Based on the actual application scenario, configure several parameters for model training and import the training dataset and test dataset paths into the corresponding locations.

[0033] S35. Define the loss function and optimization process. The loss function is used to measure the difference between the model's prediction and the true label, while the optimizer is used to update the model's parameters to minimize the loss function. Commonly used loss functions include cross-entropy loss function and mean squared error loss function. The optimization process includes gradient descent, RMSProp algorithm and Adam algorithm.

[0034] S36. Model training process: Input the training samples into the Unet model for training. Iterate and update the model parameters repeatedly to minimize the loss function and obtain a model weight file that meets the conditions for subsequent prediction tasks.

[0035] Optionally, S4 includes the following steps:

[0036] S41. Model testing process: During the testing phase of the Unet model, accuracy and intersection-over-union ratio are used as evaluation metrics for semantic segmentation to assess the model's performance. The model is run multiple times on the test set, and the average of the data is taken as the final result data.

[0037] The evaluation indicators are calculated as follows:

[0038] Accuracy:

[0039] Accuracy = n correct / n total

[0040] Where, n ncorrect n is the number of samples that are correctly classified. total This represents the total number of samples;

[0041] Intersection and Union:

[0042]

[0043] Define the diagonal coordinates of rectangles A and B as [(x... minA ,y minA ),(x maxA ,y maxA )]、[(x minB ,y minB ),(x maxB ,y maxB The diagonal coordinates of the intersection matrix are defined as [(x)]. minAB ,y minAB ),(x maxAB ,y maxAB Therefore, the formulas for calculating intersection and union are as follows:

[0044] A∩B=(x maxAB -x minAB )*(y maxAB -y minAB )

[0045] W A =x maxA -x minA

[0046] H A =y maxA -y minA

[0047] W B =x maxB -x minB

[0048] H B =y maxB -yminB

[0049] A∪B=W A *H A +W B *H B -A∩B

[0050] S42. In the optimization phase of the Unet model, the model is optimized based on the test results of the trained model.

[0051] Optionally, S5 further includes the following steps:

[0052] S51. Capture hatch cover processing: Capture the rectangular data of the hatch cover using the built-in preset hatch cover detection function of the industrial camera, including the coordinates of the four vertices of the positive matrix.

[0053] S52. Semantic segmentation processing: Initialize the hatch cover size to 512×512 and use it as input data. Import it into the Unet model, perform semantic segmentation on the input data, and obtain the hatch cover segmentation mask image.

[0054] S53. Morphological operations: Use the getStructuringElement function in the OpenCV library to create a 3x3 rectangular structuring element, and then use the morphologyEx function to perform an opening operation to remove some known noise or interference.

[0055] S54. Contour discovery and drawing: Use the findContours function in the OpenCV library to find the contours in the hatch segmentation mask and store all contour points.

[0056] S55. Sort the contours by sorting the areas of all the acquired contours, removing all contours except the one with the largest area, and use this as the output data, including the conversion relationship between partial point coordinates and overall point coordinates, to prepare for the subsequent conversion of edge positioning point coordinates.

[0057] Optionally, S6 further includes:

[0058] S61. Convex hull detection processing: The obtained maximum contour mask image and point set are subjected to convex hull detection using the convexHull function in the OpenCV library. The contour edges are refined by determining whether the contour is a convex hull.

[0059] S62. Corner detection processing: Harris corner detection is used to detect corners in the contour and returns a set of corner data. The calculation formula for Harris corner detection is as follows:

[0060] E(u,v)=∑ x,yω(x,y)[I(x+u,y+v)-I(x,y)] 2

[0061] Where E(u,v) is a matrix with the same size as the image, ω(u,v) represents a square binary window, which takes the value 1 if the pixel is inside the window and 0 otherwise, and I(x,y) represents the gray value intensity of the pixel at x,y, ranging from 0 to 255.

[0062] S63. Draw a regular rectangle. Use the boundingRect function in the OpenCV library to obtain the bounding box of the outline, draw the bounding box, and obtain the coordinates of the four corner points of the outer regular rectangle.

[0063] S64. Calculate the Euclidean distance by iterating through the set of contour corner points, taking the coordinates of the upper right corner of the circumscribed rectangle, calculating the Euclidean distance from the corner point to that point, and selecting the corner point with the smallest distance. The Euclidean distance calculation formula is as follows:

[0064]

[0065] S65. Coordinate transformation processing: The obtained corner coordinates are transformed into coordinates on the overall map using the transformation relationship between the coordinate points mentioned above, resulting in the final coordinate data.

[0066] S66. Edge positioning processing: The obtained coordinate data is plotted onto the overall image to complete the edge positioning of the hatch cover.

[0067] Compared with existing technologies, this invention provides a method for locating the edge of a port hatch cover based on Unet semantic segmentation, which has the following advantages:

[0068] This method for locating the edge of a port hatch cover based on Unet semantic segmentation trains a Unet model using processed image data, enabling the model to accurately identify the edge position of the hatch cover. The trained Unet model is then tested and optimized to improve its accuracy and generalization ability, thereby determining the set of edge points of the hatch cover. Contour extraction is used to extract the edge of the hatch cover, convex hull detection is performed and the convex hull is drawn, the set of corner points is traversed, and the minimum Euclidean distance from the edge corner point to the corner point of the rectangular hatch cover is calculated to determine the optimal corner point of the hatch cover edge, thus completing the edge localization work. This method achieves higher accuracy and efficiency, lower cost, and is suitable for widespread use. Attached Figure Description

[0069] Figure 1 This is a schematic diagram of the port hatch cover edge localization method based on Unet semantic segmentation provided in an embodiment of the present invention;

[0070] Figure 2This is a schematic diagram of the imaging principle of an industrial camera provided in an embodiment of the present invention;

[0071] Figure 3 This is a schematic diagram of the training dataset for the Unet semantic segmentation model provided in an embodiment of the present invention, wherein the first row is a schematic diagram of the original images and the second row is a schematic diagram of the training label images;

[0072] Figure 4 This is a schematic diagram illustrating the training set performance of the Unet-based semantic segmentation model provided in an embodiment of the present invention.

[0073] Figure 5 This is a schematic diagram illustrating the validation set performance of the Unet-based semantic segmentation model provided in an embodiment of the present invention.

[0074] Figure 6 This is a schematic diagram of the intersection-union ratio in the model evaluation process provided in an embodiment of the present invention;

[0075] Figure 7 This is a schematic diagram of the edge thinning effect after contour extraction provided in an embodiment of the present invention, wherein the first row is a schematic diagram of the original image, the second row is a schematic diagram of the semantic segmentation image, and the third row is a schematic diagram of the contour edge thinning effect;

[0076] Figure 8 This is a schematic diagram illustrating the edge positioning effect of a port hatch cover provided in an embodiment of the present invention. Detailed Implementation

[0077] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0078] like Figures 1-8 As shown, the present invention provides a technical solution: a method for locating the edge of a port hatch cover based on Unet semantic segmentation, comprising the following steps:

[0079] S1. Data Acquisition: Data is acquired through industrial cameras. Based on the imaging principle of industrial cameras, video or image data is transmitted to the storage unit for model training or semantic segmentation.

[0080] S11. Search and retrieve the camera information list, then open the corresponding camera according to the camera information list and obtain the camera object. Camera parameters can only be set after the camera is opened;

[0081] S12. Select either internal trigger acquisition mode or external trigger acquisition mode. Internal trigger acquisition includes continuous acquisition and single-frame acquisition, while external trigger acquisition includes soft trigger and hard trigger. The difference between the two is that the latter requires sending a trigger command to acquire image information.

[0082] S13. Open the acquisition stream;

[0083] S14. If the internal trigger acquisition mode is used, the image can be acquired by opening the acquisition stream; if the external trigger acquisition mode is used, a trigger command needs to be sent in order to acquire the image, and one trigger command acquires one image.

[0084] S15. Stream capture and image capture. Stream capture and image capture methods are divided into active stream capture and image capture via callback. If callback stream capture and image capture is used, the callback function registration operation should be placed before the operation of opening the acquisition stream;

[0085] S16. Close the acquisition stream;

[0086] S17. Turn off the camera.

[0087] S2. Data Preprocessing: To improve the accuracy and reliability of data analysis after model training, reduce the amount of data to be processed during data analysis, reduce computing resources and time costs, thereby improving data quality and readability, and better supporting decision-making and business optimization, data preprocessing is performed after data collection. This involves integrating data from different data sources, removing outliers, missing values, and redundant data, optimizing model performance, and improving the model's accuracy and generalization ability. This includes the following steps:

[0088] S21. Data Cleaning: Data cleaning is a crucial step in data preprocessing. It includes removing outliers, handling missing values, removing redundant data, and screening the original images. Data with poor lighting, blurry images, or excessively bright lighting is removed, and images that are clear and contain the hatch cover are selected for later use. The following is pseudocode for cleaning data containing missing and outliers:

[0089]

[0090]

[0091] S22. Data Augmentation: In order to solve the problem of insufficient data in port application scenarios and reduce the probability of imbalance between positive and negative samples, this embodiment of the invention uses data augmentation to amplify a small number of samples to make them as close as possible to the real distribution of data, so that the model can learn more robust features and improve the model's generalization ability and detection accuracy.

[0092] Cropping and scaling: This involves changing the size and perspective of an image by adding background around it or removing edge pixels, and by scaling it up or down. Below is pseudocode for cropping and scaling:

[0093]

[0094]

[0095] Rotation and flipping: This involves rotating an image by a certain angle to change its viewpoint and orientation, or flipping it horizontally or vertically by a certain angle to change its symmetry and orientation. Below is pseudocode for rotation and flipping:

[0096]

[0097] Color transformation processing: This involves increasing the diversity and quantity of data by altering the brightness, contrast, and saturation of an image. Below is pseudocode for color transformation:

[0098]

[0099]

[0100] Grayscale processing: Converting a color image to a grayscale image simplifies image processing and improves efficiency. This patent embodiment uses a weighted average method for grayscale processing, setting different weights to divide the RGB components into grayscale values ​​at different ratios. Since the human eye has the lowest sensitivity to blue and the highest sensitivity to green, a reasonable grayscale image is obtained by weighting the RGB components according to ratios of 0.299, 0.587, and 0.114. The calculation formula and pseudocode are shown below:

[0101] Gray(i,j)=0.299*R(i,j)+0.578*G(i,j)+0.114*B(i,j)

[0102]

[0103]

[0104] S3, Training the Unet model

[0105] The method for locating the edge of a port hatch cover based on Unet semantic segmentation provided in this invention uses the Unet semantic segmentation network as the main component and trains the Unet network. The main steps are as follows:

[0106] S31. Data annotation and processing: Select 1000 prepared images including hatch covers, with a size of 512×512. Use Labelme annotation software to create a dataset, customize label names, mark the hatch covers in the image data, save the annotation results, generate XML label data, and convert it into the corresponding Mask image data.

[0107] S32. Model processing: Construct a Unet model suitable for semantic segmentation of hatch covers. The network structure of the model includes an encoder and a decoder. The encoder is responsible for extracting the feature information of the image, while the decoder restores the feature information to the same size as the original image by upsampling. The encoder and decoder pass information through skip connections to help the decoder better restore the detailed information.

[0108] S33. Divide the dataset into training and validation sets according to a set ratio of 9:1 for training and validating the model.

[0109] S34. Configure parameter processing: Based on the actual application scenario, configure several parameters for model training and import the training dataset and test dataset paths into the corresponding locations.

[0110] S35. Define the loss function and optimization process. The loss function measures the difference between the model's prediction and the true label, while the optimizer updates the model's parameters to minimize the loss function. Commonly used loss functions include cross-entropy loss and mean squared error loss. Optimization processes include gradient descent, RMSProp algorithm, and Adam algorithm. Considering the specific needs of the model's application scenario, this embodiment of the invention uses a binary cross-entropy loss function and the Adam optimization algorithm. The corresponding algorithm calculation process is as follows:

[0111] Binary cross-entropy loss function: Primarily used for binary classification problems, it combines the sigmoid activation function and the binary cross-entropy (BCE) loss function. Its main working principle and calculation formula are as follows:

[0112] The probability distribution predicted by the model after passing through the sigmoid activation function is calculated.

[0113] Calculate the binary cross-entropy (BCE) loss;

[0114] Combining the two, we arrive at the final loss.

[0115] Loss=-y*log(y)-(1-y)*log(1-y)

[0116] Where y is the probability distribution predicted by the model, that is, the output after passing through the sigmoid function, and y is the true label, which takes the value 0 or 1.

[0117] Adam optimization algorithm: This is an adaptive learning rate optimization algorithm used to solve the learning rate adjustment problem in gradient descent. It combines the ideas of Momentum and RMSprop. In each iteration, it updates each parameter based on the first and second moment estimates of the gradient. The first moment estimate helps accelerate the gradient in areas with small gradients, while the second moment estimate helps suppress gradient noise. The main calculation formulas are as follows:

[0118] m t := beta1*m t-1 +(1-beta1)*g (1)

[0119] v t := beta2*v t-1 +(1-beta2)*g*g (2)

[0120]

[0121] Formula (1) mainly calculates the first-order exponential smoothing value of the historical gradient, which is used to obtain the gradient value with momentum; Formula (2) mainly calculates the first-order exponential smoothing value of the square of the historical gradient, which is used to obtain the learning rate weight parameter of each weight parameter; Formula (3) mainly calculates the variable update value, which is the first-order exponential smoothing value that is proportional to the historical gradient and the first-order exponential smoothing value that is inversely proportional to the square of the historical gradient.

[0122] S36. Model training process: Input the training samples into the Unet model for training. Iterate and update the model parameters to minimize the loss function and obtain a qualified model weight file for subsequent prediction tasks. During this process, the training status and performance of the model can be monitored through output logs to achieve model application in port scenarios.

[0123] S4, Model Testing and Optimization

[0124] The method for locating the edge of a port hatch cover based on Unet semantic segmentation provided in this invention uses the Unet semantic segmentation model as the main component and tests and optimizes the Unet model, including the following steps:

[0125] S41. Model Testing: During the testing phase of the Unet model, accuracy and intersection-over-union ratio (IoU) are used as evaluation metrics for semantic segmentation to assess the model's performance. The model is run multiple times on the test set, and the average of the data is taken as the final result. The evaluation metrics are calculated as follows:

[0126] Accuracy:

[0127] Accuracy = n correct / n total

[0128] Where, n ncorrect n is the number of samples that are correctly classified. total This represents the total number of samples.

[0129] Intersection and Union:

[0130]

[0131] Define the diagonal coordinates of rectangles A and B as [(x... minA ,y minA ),(x maxA ,y maxA )]、[(x minB ,y minB ),(x maxB ,y maxB The diagonal coordinates of the intersection matrix are defined as [(x)]. minAB ,y minAB ),(x maxAB ,y maxAB Therefore, the formulas for calculating intersection and union are as follows:

[0132] A∩B=(x maxAB -x minAB )*(y maxAB -y minAB )

[0133] W A =x maxA -x minA

[0134] H A =y maxA -y minA

[0135] W B =x maxB -x minB

[0136] H B =y maxB -y minB

[0137] A∪B=W A *H A +W B *H B -A∩B

[0138] S42. Model Optimization: In the optimization phase of the Unet model, the model is optimized based on the test results of the trained model. This embodiment uses batch normalization and data augmentation to improve the performance of the Unet model. Data augmentation is as described above. Batch normalization mainly avoids the problems of gradient vanishing and exploding by normalizing the data in each batch to the same scale. The main calculation formula is as follows:

[0139]

[0140] Where, x i+1 μ represents the normalized data. B σ represents the mean of this batch of samples. B The variance of the sample is represented by the parameter γ. What is special is that the batch normalization used in this embodiment of the invention adds a stretching parameter γ and an offset parameter β to the basic data.

[0141] S5, Semantic Segmentation

[0142] The method for port hatch cover edge localization based on Unet semantic segmentation provided in this invention uses real-time data from industrial cameras as the main source and performs semantic segmentation on the hatch covers in the data. The main contents are as follows:

[0143] S51. Capture hatch cover processing: Capture the rectangular data of the hatch cover using the built-in preset hatch cover detection function of the industrial camera, including the coordinates of the four vertices of the positive matrix.

[0144] S52. Semantic segmentation processing: Initialize the hatch cover size to 512×512 and use it as input data. Import it into the Unet model, perform semantic segmentation on the input data, and obtain the hatch cover segmentation mask image.

[0145] S53. Morphological operations: Use the getStructuringElement function in the OpenCV library to create a 3x3 rectangular structuring element, and then use the morphologyEx function to perform an opening operation to remove some known noise or interference.

[0146] S54. Contour discovery and drawing: Use the findContours function in the OpenCV library to find the contours in the hatch segmentation mask and store all contour points.

[0147] S55. Sort the contours by sorting the areas of all the acquired contours, removing all contours except the one with the largest area, and use this as the output data, including the conversion relationship between partial point coordinates and overall point coordinates, to prepare for the subsequent conversion of edge positioning point coordinates.

[0148] S6, Edge Positioning

[0149] The method for locating the edge of a port hatch cover based on Unet semantic segmentation provided in this invention uses the Unet model segmentation mask image as the main component and refines the hatch cover outline in the image. The main contents are as follows:

[0150] S61. Convex hull detection processing: The obtained maximum contour mask image and point set are subjected to convex hull detection using the convexHull function in the OpenCV library. The contour edges are refined by determining whether the contour is a convex hull.

[0151] S62. Corner detection processing: Harris corner detection is used to detect corners in the contour and returns a set of corner data. The calculation formula for Harris corner detection is as follows:

[0152] E(u,v)=∑ x,y ω(x,y)[I(x+u,y+v)-I(x,y)] 2

[0153] Where E(u,v) is a matrix with the same size as the image, ω(u,v) represents a square binary window, which takes the value 1 if the pixel is inside the window and 0 otherwise, and I(x,y) represents the gray value intensity of the pixel at x,y, ranging from 0 to 255.

[0154] S63. Draw a regular rectangle. Use the boundingRect function in the OpenCV library to obtain the bounding box of the outline, draw the bounding box, and obtain the coordinates of the four corner points of the outer regular rectangle.

[0155] S64. Calculate the Euclidean distance by iterating through the set of contour corner points, taking the coordinates of the upper right corner of the circumscribed rectangle, calculating the Euclidean distance from the corner point to that point, and selecting the corner point with the smallest distance. The Euclidean distance calculation formula is as follows:

[0156]

[0157] S65. Coordinate transformation processing: The obtained corner coordinates are transformed into coordinates on the overall map using the transformation relationship between the coordinate points mentioned above, resulting in the final coordinate data.

[0158] S66. Edge positioning processing: The obtained coordinate data is plotted onto the overall image to complete the edge positioning of the hatch cover.

[0159] The present invention has been described in detail above. However, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, any modifications or improvements that do not depart from the spirit of the present invention are within the scope of protection of the present invention.

Claims

1. A method for locating the edge of a port hatch cover based on Unet semantic segmentation, characterized in that: The method includes the following steps: S1. Data Acquisition: First, it is necessary to collect image data of the port hatch cover, which can be obtained by capturing images or videos with a camera. S2. Data preprocessing: Preprocess the acquired image data, including noise removal, contrast enhancement, and data augmentation operations, to improve image quality; S3. Training the Unet model: Use the processed image data to train the Unet model so that the model can accurately identify the edge position of the hatch cover. S4. Model Testing and Optimization: Test and optimize the trained Unet model to improve its accuracy and generalization ability. S5. Semantic segmentation: The trained Unet model is used to perform semantic segmentation on the new port hatch cover image to determine the set of edge points of the hatch cover. S6. Edge localization: The edge of the hatch cover is extracted using the contour extraction method, the convex hull is detected and drawn, the set of corner points is traversed, and the minimum Euclidean distance from the edge corner point to the corner point of the rectangular hatch cover is calculated to determine the optimal corner point of the hatch cover edge and complete the edge localization work. S5 further includes the following steps: S51. Capture hatch cover processing: Capture the rectangular data of the hatch cover using the built-in preset hatch cover detection function of the industrial camera, including the coordinates of the four vertices of the positive matrix. S52. Semantic segmentation processing: Initialize the hatch cover size to 512×512 and use it as input data. Import it into the Unet model, perform semantic segmentation on the input data, and obtain the hatch cover segmentation mask image. S53. Morphological operations: Use the getStructuringElement function in the OpenCV library to create a 3x3 rectangular structuring element, and then use the morphologyEx function to perform an opening operation to remove some known noise or interference. S54. Contour discovery and drawing: Use the findContours function in the OpenCV library to find the contours in the hatch segmentation mask and store all contour points. S55. Sort the contours by sorting the areas of all the acquired contours, removing all contours except the one with the largest area, and use it as the output data, including the conversion relationship between partial point coordinates and overall point coordinates, to prepare for the subsequent conversion of edge positioning point coordinates. S6 further includes: S61. Convex hull detection processing: The obtained maximum contour mask image and point set are subjected to convex hull detection using the convexHull function in the OpenCV library. The contour edges are refined by determining whether the contour is a convex hull. S62. Corner detection processing: Harris corner detection is used to detect corners in the contour and returns a set of corner data. The calculation formula for Harris corner detection is as follows: ； in, It is a matrix with the same size as the image. This represents a rectangular binary window. If a pixel is inside the window, the value is 1; otherwise, it is 0. This represents the intensity of the pixel grayscale value at x and y, ranging from 0 to 255. S63. Draw a regular rectangle. Use the boundingRect function in the OpenCV library to obtain the bounding box of the outline, draw the bounding box, and obtain the coordinates of the four corner points of the outer regular rectangle. S64. Calculate the Euclidean distance by iterating through the set of contour corner points, taking the coordinates of the upper right corner of the circumscribed rectangle, calculating the Euclidean distance from the corner point to that point, and selecting the corner point with the smallest distance. The Euclidean distance calculation formula is as follows: ; S65. Coordinate transformation processing: The obtained corner coordinates are transformed into coordinates on the overall map using the transformation relationship between the coordinate points mentioned above, resulting in the final coordinate data. S66. Edge positioning processing: The obtained coordinate data is plotted onto the overall image to complete the edge positioning of the hatch cover.

2. The method for locating the edge of a port hatch cover based on Unet semantic segmentation according to claim 1, characterized in that: S1 further includes: S11. Search for and retrieve the camera information list, then open the corresponding camera according to the camera information list and retrieve the camera object; S12. Select either internal trigger acquisition mode or external trigger acquisition mode. Internal trigger acquisition includes continuous acquisition and single-frame acquisition, while external trigger acquisition includes soft trigger and hard trigger. S13. Open the acquisition stream; S14. If the internal trigger acquisition mode is used, the image can be acquired by opening the acquisition stream; if the external trigger acquisition mode is used, a trigger command is sent. S15. Capture stream and capture images. There are two ways to capture stream and capture images: active capture stream and capture stream and capture stream and capture. If you use the callback capture stream and capture method, the callback function registration operation should be placed before the operation of opening the capture stream. S16. Close the acquisition stream; S17. Turn off the camera.

3. The method for locating the edge of a port hatch cover based on Unet semantic segmentation according to claim 1, characterized in that: S2 further includes: S21. Data cleaning: remove outliers, handle missing values, remove redundant data, screen the original images, remove data with poor lighting conditions, blurry images, or excessive lighting conditions, and select some clear images that contain hatch covers for backup. S22. Perform data enhancement processing.

4. The method for locating the edge of a port hatch cover based on Unet semantic segmentation according to claim 3, characterized in that: The enhancement process includes: Cropping and scaling processes change the size and perspective of an image by adding background around it or removing some edge pixels, and by scaling the image up or down. Rotation and flipping processing changes the viewpoint and orientation of an image by rotating it by a certain angle, and changes its symmetry and orientation by flipping it horizontally or vertically by a certain angle. Color transformation processing increases the diversity and quantity of data by changing the brightness, contrast, and saturation of an image; Grayscale conversion simplifies image processing by converting color images to grayscale.

5. The method for locating the edge of a port hatch cover based on Unet semantic segmentation according to claim 1, characterized in that: S3 further includes: S31. Data annotation and processing: Select 1000 prepared images including hatch covers, with a size of 512×512. Use Labelme annotation software to create a dataset, customize label names, mark the hatch covers in the image data, save the annotation results, generate XML label data, and convert it into the corresponding Mask image data. S32. Constructing a model for semantic segmentation of hatch covers. The network structure of the model includes an encoder and a decoder. The encoder is responsible for extracting the feature information of the image, and the decoder restores the feature information to the same size as the original image by upsampling. The encoder and decoder transmit information through skip connections. S33. Divide the dataset into training and validation sets according to a set ratio of 9:1 for training and validating the model. S34. Configure parameter processing: Based on the actual application scenario, configure several parameters for model training and import the training dataset and test dataset paths into the corresponding locations. S35. Define the loss function and optimization process. The loss function is used to measure the difference between the model's prediction and the true label, while the optimizer is used to update the model's parameters to minimize the loss function. Commonly used loss functions include cross-entropy loss function and mean squared error loss function. The optimization process includes gradient descent, RMSProp algorithm and Adam algorithm. S36. Model training process: Input the training samples into the Unet model for training. Iterate and update the model parameters repeatedly to minimize the loss function and obtain a model weight file that meets the conditions for subsequent prediction tasks.

6. The method for locating the edge of a port hatch cover based on Unet semantic segmentation according to claim 1, characterized in that: S4 includes the following steps: S41. Model testing process: During the testing phase of the Unet model, accuracy and intersection-over-union ratio are used as evaluation metrics for semantic segmentation to assess the model's performance. The model is run multiple times on the test set, and the average of the data is taken as the final result data. The evaluation indicators are calculated as follows: Accuracy: ； in, The number of samples that were correctly classified. This represents the total number of samples; Intersection and Union: ； Define the diagonal coordinates of rectangle A and rectangle B as follows: , Define the diagonal coordinates of the intersection matrix as... Therefore, the formulas for calculating the intersection and union are as follows: ； S42. In the optimization phase of the Unet model, the model is optimized based on the test results of the trained model.

Images