A method and system for detecting abnormalities in the connection of ship-side containers based on machine vision

Abstract

The present invention discloses a method and system for detecting abnormal connection of ship - side containers based on machine vision. The method includes: obtaining the opening and closing state of the quay crane spreader and the monitoring video image at the quay crane spreader; based on the opening and closing state of the quay crane spreader, detecting the monitoring video image at the quay crane spreader based on an object detection model to obtain an abnormal detection area; analyzing the abnormal detection area based on the pyramid - layered LK optical flow method to obtain the lifting height of the working container or the connected container; and judging whether there is an abnormal connection of the container according to the lifting height of the working container or the connected container. The technical solution provided by the present invention uses object detection models and optical flow method analysis to be able to detect in real time whether the spreader is normally hanging the working container and whether there is a connected container, realizes high - precision connection detection, improves the reliability and accuracy of detection, and ensures the operation safety of the quay crane during the lifting process.

Description

Technical Field

[0001] The present invention relates to the field of container detection, and particularly to a method and system for detecting abnormal connection of ship-side containers based on machine vision. Background Art

[0002] With the continuous increase in container throughput, the unloading operations of quay cranes are becoming increasingly intensive. Since containers are usually stored in a highly stacked manner on ships with small mutual spacing, abnormal connection phenomena may occur during the lifting operation of the spreader. Even in some cases, the working container may be accidentally connected to the container below, thus bringing potential safety risks.

[0003] Currently, sensor monitoring methods are usually used to determine whether there is abnormal connection of containers. This method has a complex circuit, is easily affected by harsh working environments, and is limited by factors such as the center-of-gravity offset of heavy containers and spreader vibration, which easily leads to false alarms and poor actual detection effects. Summary of the Invention

[0004] In view of the above deficiencies in the current technology, the present invention provides a method for detecting abnormal connection of ship-side containers based on machine vision. Based on quay crane lifting image data and a target detection model, an abnormal detection area is obtained, and then through optical flow method analysis, the lifting heights of the working container and the connected container are obtained, thereby determining whether abnormal connection has occurred.

[0005] To achieve the above object, the embodiments of the present invention adopt the following technical solutions:

[0006] A method for detecting abnormal connection of ship-side containers based on machine vision, comprising the following steps:

[0007] Obtain the opening and closing state of the quay crane spreader and the monitoring video image at the quay crane spreader;

[0008] Based on the opening and closing state of the quay crane spreader, detect the monitoring video image at the quay crane spreader based on the target detection model to obtain an abnormal detection area;

[0009] Analyze the abnormal detection area based on the pyramid hierarchical LK optical flow method to obtain the lifting height of the working container or the connected container;

[0010] Based on the lifting height of the working container or the connected container, determine whether abnormal connection of the container has occurred.

[0011] According to one aspect of the present invention, the obtaining the opening and closing state of the quay crane spreader and the monitoring video image at the quay crane spreader includes:

[0012] The PLC of the quay crane controls the spreader to unlock or lock;

[0013] Obtain the control signal of the PLC and judge the opening and closing state of the quay crane spreader;

[0014] Use vision equipment to collect the monitoring video images at the quay crane spreader.

[0015] According to one aspect of the present invention, the monitoring video of the quay crane spreader is detected based on the object detection model according to the opening and closing state of the quay crane spreader, and obtaining the abnormal detection area includes:

[0016] When the spreader is in the unlocked state, the abnormal detection area is the spreader and the working container below the spreader;

[0017] When the spreader is in the locked state, the abnormal detection area is the working container below the spreader and the hooked container below the working container.

[0018] According to one aspect of the present invention, the detection of the monitoring video image at the quay crane spreader based on the object detection model includes:

[0019] Use the YOLO, YOLOv8 or improved YOLOv8 object detection model to detect the monitoring video image at the quay crane spreader and identify the spreader, working container and hooked container.

[0020] According to one aspect of the present invention, the detection of the monitoring video image at the quay crane spreader based on the object detection model includes:

[0021] Construct a YOLOv8 object detection model to detect and locate the spreader, working container and hooked container.

[0022] According to one aspect of the present invention, the improved YOLOv8 model includes:

[0023] Backbone layer, perform feature extraction on the monitoring video image based on the convolutional module, C2f module, SPPF module and C2PFA self-attention module;

[0024] Neck layer, use the features extracted by the Backbone layer, and perform multi-scale feature fusion based on upsampling, feature splicing, convolutional module, C2f module, and FPN+PAN structure to obtain fused features;

[0025] Head layer, according to the fused features, use multiple detection heads to detect and locate the areas where the spreader, working container and hooked container are located.

[0026] According to one aspect of the present invention, analyzing the abnormal detection area based on the pyramid hierarchical LK optical flow method to obtain the lifting height of the working container or the hooked container includes:

[0027] Perform grayscale processing and multiple downsamplings on the anomaly detection area to obtain a pyramid image sequence;

[0028] For two consecutive sampled images, perform optical flow estimation from top to bottom;

[0029] Utilize the optical flow estimation result to obtain the lifting height of the working container or the hooked container.

[0030] According to one aspect of the present invention, the performing optical flow estimation from top to bottom on two consecutive sampled images includes:

[0031] Calculate the gray-scale gradients in the x and y axis directions and the gray-scale gradient on the time axis at a certain point in the image;

[0032] According to the optical flow constraint equation, use the least squares method to solve the optimal solution of the optical flow of two consecutive sampled images.

[0033] According to one aspect of the present invention, the determining whether the container is abnormally hooked based on the lifting height of the working container or the hooked container includes:

[0034] When the spreader is in the unlocked state, if the lifting height of the working container exceeds a preset height threshold, it is determined that an abnormal hook-up has occurred;

[0035] When the spreader is in the locked state, if the lifting height of the hooked container exceeds a preset height threshold, it is determined that an abnormal hook-up has occurred.

[0036] A ship-side container hook-up anomaly detection system based on machine vision, based on the above-mentioned ship-side container hook-up anomaly detection method based on machine vision, includes:

[0037] An acquisition module for obtaining the open / closed state of the quay crane spreader and the monitoring video image at the quay crane spreader;

[0038] A detection module for detecting the monitoring video image at the quay crane spreader based on the target detection model according to the open / closed state of the quay crane spreader to obtain the anomaly detection area;

[0039] An optical flow analysis module for analyzing the anomaly detection area based on the pyramid hierarchical LK optical flow method to obtain the lifting height of the working container or the hooked container;

[0040] A judgment module for determining whether the container is abnormally hooked according to the lifting height of the working container or the hooked container.

[0041] Advantages of the implementation of the present invention:

[0042] The present invention provides a method for detecting abnormal connection of ship - side containers based on machine vision. Based on the real - time collected quay - crane lifting image data and the target detection model, the regions where the spreader, the working container, and the connected container are located are obtained. Then, the gray - scale change in consecutive frame images is analyzed by the optical flow method to obtain the lifting heights of the working container and the connected container, thereby determining whether abnormal connection occurs.

[0043] This method can detect in real - time whether the spreader is normally mounted on the working container and determine whether there is a connected container below the working container. According to the detection results, the system can control the quay - crane spreader to pause lifting and issue an alarm, achieving high - precision connection detection, improving the reliability and accuracy of detection, significantly enhancing the safety monitoring ability during the container - lifting process, and ensuring the operation safety of the quay - crane during lifting. Brief Description of the Drawings

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0045] Figure 1 It is a flowchart of a method for detecting abnormal connection of ship - side containers based on machine vision according to the present invention;

[0046] Figure 2 It is a schematic diagram of the quay - crane structure according to the present invention;

[0047] Figure 3 It is a schematic diagram of the open - lock state of the quay - crane spreader according to the present invention;

[0048] Figure 4 It is a general processing flowchart of a method for detecting abnormal connection of ship - side containers based on machine vision according to the present invention;

[0049] Figure 5 It is a schematic diagram of the improved YOLOv8 model architecture according to Embodiment 2 of the present invention.

[0050] Reference Signs: 1. Front girder of the quay - crane; 2. Vision device; 3. Spreader; 4. PLC; 5. Server; 6. Electrical room; 7. Working container; 8. Connected container. Detailed Embodiments

[0051] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0052] Embodiment 1

[0053] As Figure 1 shown, a method for detecting abnormal connection of ship - side containers based on machine vision includes the following steps:

[0054] S1: Obtain the opening and closing state of the quay crane spreader and the monitoring video image at the quay crane spreader.

[0055] Specifically, step S1 includes:

[0056] The PLC of the quay crane controls the spreader to unlock or lock;

[0057] Obtain the control signal of the PLC and judge the opening and closing state of the quay crane spreader;

[0058] Use the vision device to collect the monitoring video image at the quay crane spreader.

[0059] In practical applications, as Figure 2 shown, vision devices can be deployed at multiple positions on the front girder of the quay crane, such as on the left and right sides of the front girder and the lower surface of the connecting beam, etc. Real - time image acquisition of the spreader and its operation target is carried out from multiple angles, which can effectively avoid the detection blind area caused by cargo occlusion in a single perspective. The vision device uses a high - definition camera, and the installation angle is in the top - down or side - view direction, covering the combined area of the spreader and the container.

[0060] There is a spreader and an electrical room on the quay crane. A server and a PLC are deployed inside the electrical room. The server is connected to the vision device and the PLC. The server and the vision device are connected through optical fiber or 5G communication for real - time transmission of image data.

[0061] Through the vision devices deployed at multiple positions on the front girder, the working state of the spreader is monitored in real time. By obtaining the side view of the spreader, the occlusion caused by the perspective problem is avoided, so as to ensure a clear picture. At the same time, the video data collected by the vision device will be transmitted to the server.

[0062] [[ID=3,7]]The PLC transmits the signal for controlling the spreader to the server, and the server analyzes the current working state of the spreader, whether it is in the unlocked or locked state, according to the received PLC signal.

[0063] S2: According to the opening and closing state of the quay crane spreader, detect the monitoring video image at the quay crane spreader based on the target detection model to obtain the abnormal detection area.

[0064] As Figure 3 shown, when the quay crane spreader is in different states, the abnormal areas to be detected are also different. Specifically:

[0065] (1) When the spreader is in the unlocked state, it is necessary to check that the spreader is unlocked normally, that is, whether there is a container being hooked below. Therefore, the abnormal detection area is the spreader and the working container below the spreader. At this time, analyze the real-time video image through the pre-trained target detection model, locate the positions of the spreader and the working container, and generate the detection bounding boxes of the spreader and the working container.

[0066] (2) When the spreader is in the locked state, it is necessary to detect whether there is an adjacent container being hooked below the currently lifted working container. Therefore, the abnormal detection area is the working container below the spreader and the hooked container below the working container. At this time, analyze the real-time video image through the pre-trained target detection model, locate the positions of the working container and the hooked container, and generate the detection bounding boxes of the working container and the hooked container.

[0067] In practical applications, target detection models such as YOLO, YOLOv8, other improved versions of YOLO, SSD, Faster R-CNN, etc. can be used to detect the monitoring video image at the quay crane spreader, identify the spreader, the working container, and the hooked container, and frame the detection area.

[0068] For example, 3 YOLOv8 detection models can be constructed to detect and locate the spreader, the working container below the spreader, and the hooked container below the working container respectively.

[0069] Specifically, the YOLOv8 model architecture includes:

[0070] (1) The Backbone layer extracts the features of the monitoring video image based on the convolutional module, C2f module, and SPPF module.

[0071] (2) Neck layer: Using the features extracted by the Backbone layer, it performs multi-scale feature fusion based on upsampling, feature concatenation, convolutional modules, C2f modules, and the FPN+PAN structure to obtain fused features. The Neck layer adopts the FPN+PAN structure, where FPN represents the Feature Pyramid Network and PAN represents the Path Aggregation Network. Through bidirectional feature transfer, it achieves sufficient fusion of feature maps at different scales, retaining both high-level semantic information and using low-level localization information, thereby significantly improving the performance of object detection.

[0072] (3) Head layer: Based on the fused features, it detects and locates the regions where the objects are located.

[0073] The training of the YOLOv8 model includes:

[0074] (1) Collect videos of the daily operations of the spreader, and calibrate the positions of the spreader, working container, and hooked container in the videos to generate a training database.

[0075] (2) Divide the training database into a training set and a test set at a ratio of 8:2. Use the training set to train the YOLOv8 model to obtain an object detection model, and evaluate its performance through the test set.

[0076] (3) By adjusting the hyperparameters of the model, select the object detection model with the best detection effect as the final version in the actual working scenario.

[0077] When applying the YOLOv8 model, according to the working state of the spreader, given the coordinate positions of the spreader and the working container in the image, crop the approximate regions where the spreader, working container, and hooked container are located in the image as the detection regions, and use the object detection model to detect and locate the positions of the spreader, working container, and hooked container respectively.

[0078] S3: Analyze the anomaly detection region based on the pyramid hierarchical LK optical flow method to obtain the lifting height of the working container or the hooked container.

[0079] In practical applications, the pyramid hierarchical LK optical flow method can be used to obtain the moving distance of the object in the image, that is, the lifting height of the working container or the hooked container.

[0080] Specifically, step S3 includes:

[0081] S31: Perform grayscale processing and multiple downsamplings on the anomaly detection region to obtain a pyramid image sequence.

[0082] Gray-scale the anomaly detection region located by the target detection model in step S2; then apply a 5×5 Gaussian kernel with a standard deviation of and perform multiple downsamplings with a stride of 2, each time reducing the image to half of its original size. Let and be the gray-scale values of a certain point on the image before and after sampling respectively.

[0083] Among them, the Gaussian kernel represents the Gaussian kernel similarity between the m-th vector and the n-th vector in the input space; l is the number of downsamplings; x and y are the pixel coordinate positions.

[0084] S32: Estimate the optical flow from top to bottom for two consecutive sampled images.

[0085] Specifically, step S32 includes:

[0086] S321: Calculate the gray-scale gradients of the image in the x and y axis directions and the gray-scale gradient on the time axis at a certain point.

[0087] Starting from the topmost image and using two consecutive sampled images , calculate the image gradients.

[0088]

[0089] Among them, respectively represent the gray-scale gradients of the image in the direction along the axis, and represents the gray-scale gradient of on the time axis.

[0090] S322: According to the optical flow constraint equation, use the least squares method to solve the optimal solution of the optical flow of two consecutive sampled images.

[0091] To solve the optical flow of a certain point in the image, use the 3×3 neighborhood window of this point as the constraint condition to solve the overdetermined optical flow constraint equation . Among them, represents the gradient values of all points in the neighborhood window in the X axis direction.

[0092]

[0093] Since the number of equations is much larger than the number of unknowns, the least squares method needs to be used to solve the optimal solution. Define the sum of squared residuals , where d is the optical flow. Take the derivative of E with respect to d and set the derivative to zero, we get , the solution is . Use the displacement of the upper layer , and initialize the displacement of the lower layer image : And correct the initialized displacement:

[0094]

[0095] S33: Using the optical flow estimation result, obtain the lifting height of the working container or the connected container.

[0096] The moving distance of the target in the image is the optical flow value of the final original image, and the optical flow value of the final original image is the superposition of the segmented optical flow of all layers :

[0097] Finally, according to the optical flow obtain the lifting height of the working container or the connected container.

[0098] S4: According to the lifting height of the working container or the connected container, determine whether the container is abnormally connected.

[0099] Specifically, step S4 includes: (1) When the spreader is in the unlocked state, if the lifting height of the working container exceeds the preset height threshold, it is determined that an abnormal connection has occurred.

[0100] According to the quay crane operation experience, a height threshold for the working container can be set in advance. When the spreader is in the unlocked state, the working container should stay in place without lifting. Therefore, when it is detected that the working container has a certain lifting height exceeding the height threshold, it can be determined that the working container has an abnormal connection.

[0101] (2) When the spreader is in the locked state, if the lifting height of the connected container exceeds the preset height threshold, it is determined that an abnormal connection has occurred.

[0102] According to the quay crane operation experience, a height threshold for the connected container can be set in advance. When the spreader is in the locked state, the working container rises with the spreader, while the lower connected container should stay in place without lifting. Therefore, when it is detected that the connected container has a certain lifting height exceeding the height threshold, it can be determined that the connected container has an abnormal connection.

[0103] When it is determined that there is an abnormal connection, the server will send a stop signal to the PLC. After receiving it, the PLC will automatically pause the rising action of the spreader. At the same time, the server will send an alarm signal to notify the staff to handle the abnormal situation in time. If no abnormality is found and the spreader or the working container has reached the predetermined height, the operation will proceed normally.

[0104] Figure 4 The overall processing flow of this method is shown as follows. In practical applications, the image data of the quay crane's hoisting operation can be collected in real time for detection to determine whether there is an abnormal connection, facilitating timely handling.

[0105] The beneficial effects of this embodiment are as follows:

[0106] Based on the image data of the quay crane's hoisting operation collected in real time and the target detection model, this method obtains the regions where the spreader, the working container, and the connected container are located. Then, by analyzing the gray-scale changes in consecutive frame images through the optical flow method, the lifting heights of the working container and the connected container are obtained, thereby determining whether there is an abnormal connection.

[0107] This method can detect in real time whether the spreader is normally attached to the working container and determine whether there is a connected container below the working container. According to the detection results, the system can control the quay crane spreader to suspend lifting and issue an alarm, achieving high-precision connection detection, improving the reliability and accuracy of detection, and ensuring the operation safety of the quay crane during hoisting.

[0108] Embodiment 2

[0109] As Figure 1 shown, a method for detecting abnormal connection of ship-side containers based on machine vision includes the following steps:

[0110] S1: Obtain the opening and closing state of the quay crane spreader and the monitoring video image at the quay crane spreader.

[0111] Specifically, step S1 includes:

[0112] The PLC of the quay crane controls the spreader to unlock or lock;

[0113] Obtain the control signal of the PLC and determine the opening and closing state of the quay crane spreader;

[0114] Use vision equipment to collect the monitoring video image at the quay crane spreader.

[0115] In practical applications, as Figure 2 shown, vision equipment can be deployed at multiple positions on the front girder of the quay crane, such as on the left and right sides of the front girder and the lower surface of the connecting beam, etc., to collect real-time images of the spreader and its working target from multiple angles, effectively avoiding the detection blind area caused by cargo occlusion in a single perspective. The vision equipment uses high-definition cameras, and the installation angle is in the top view or side view direction, covering the combined area of the spreader and the container.

[0116] The quay crane is equipped with a spreader and an electrical room. Inside the electrical room, a server and a PLC are deployed. The server is connected to the vision device and the PLC. The server and the vision device are connected through optical fiber or 5G communication for real-time transmission of image data.

[0117] Through the vision devices deployed in multiple directions on the front girder, the working state of the spreader is monitored in real time. By obtaining the side view of the spreader, the occlusion caused by the perspective problem is avoided, thus ensuring a clear picture. At the same time, the video data collected by the vision device is transmitted to the server.

[0118] The PLC transmits the signal for controlling the spreader to the server. The server analyzes the current working state of the spreader, whether it is in the unlocked or locked state, according to the received PLC signal.

[0119] S2: According to the unlocking and locking state of the quay crane spreader, the monitoring video image at the quay crane spreader is detected based on the target detection model to obtain the abnormal detection area.

[0120] As Figure 3 shown, when the quay crane spreader is in different states, the abnormal areas to be detected are also different. Specifically:

[0121] (1) When the spreader is in the unlocked state, it is necessary to check whether the spreader unlocks normally, that is, whether there is a hooked working container below. Therefore, the abnormal detection area is the spreader and the working container below the spreader. At this time, the real-time video image is analyzed by the pre-trained target detection model to locate the positions of the spreader and the working container, and generate the detection bounding boxes of the spreader and the working container.

[0122] (2) When the spreader is in the locked state, it is necessary to detect whether there is a hooked adjacent container below the currently lifted working container. Therefore, the abnormal detection area is the working container below the spreader and the hooked container below the working container. At this time, the real-time video image is analyzed by the pre-trained target detection model to locate the positions of the working container and the hooked container, and generate the detection bounding boxes of the working container and the hooked container.

[0123] In practical applications, target detection models such as YOLO, YOLOv8, other improved versions of YOLO, SSD, Faster R-CNN, etc. can be used to detect the monitoring video image at the quay crane spreader, identify the spreader, the working container and the hooked container, and frame the detection area.

[0124] For example, an improved YOLOv8 detection model can be constructed to enhance the model's perception ability of the global image context, and at the same time detect and locate the spreader, the working container below the spreader and the hooked container below the working container.

[0125] Specifically, as Figure 5As shown, the improved YOLOv8 model architecture includes:

[0126] (1) Backbone layer, which extracts the features of the monitored video image based on the convolutional module, C2f module, SPPF module, and C2PFA (Cross Stage Pixelfocused attention) self-attention module. The PFA module uses a dual-path design, where one path uses Sliding Window Attention to achieve fine perception of the target center; the other path uses Pooling attention to simulate the ignored global information; at the same time, the softmax module outputs the attention weights according to the results of the two paths; it can better process multi-scale information, thereby improving the feature extraction ability of the model.

[0127] The Backbone layer of the original YOLOv8 model can only gradually analyze the image content through layer-by-layer convolution operations, while this method improves the Backbone layer by adding the C2PFA module, which can effectively perceive global context information and improve the feature extraction ability of the model.

[0128] (2) Neck layer, which uses the features extracted by the Backbone layer to perform multi-scale feature fusion based on upsampling, feature concatenation, convolutional module, C2f module, and FPN+PAN structure to obtain fused features. The Neck layer adopts the FPN+PAN structure, where FPN represents the Feature Pyramid Network, and PAN represents the PathAggregation Network. Through bidirectional feature transfer, it realizes the full fusion of feature maps at different scales, retaining both the semantic information of the high layer and using the localization information of the low layer, thereby significantly improving the performance of object detection.

[0129] (3) Head layer, which detects and locates the areas where the spreader, working container, and hooked container are located simultaneously according to the fused features using multiple detection heads. In the design of the multi-detection head structure, a multi-channel parallel architecture is adopted to process different object detection tasks respectively, improving the localization accuracy of the model in object detection and achieving efficient recognition of multiple objects.

[0130] The Head layer of the original YOLOv8 model only includes one detection head and can only detect a single target. However, this method improves the Head layer by setting multiple detection heads in parallel, which can detect multiple targets simultaneously. The model is more concise and can also greatly reduce the computational workload.

[0131] The training of the improved YOLOv8 model includes:

[0132] (1) Collect the videos of the daily operations of the lifting appliance, and calibrate the positions of the lifting appliance, the working container, and the connected container in the videos to generate a training database.

[0133] (2) Divide the training database into a training set and a test set according to the ratio of 8:2. Use the training set to train the improved YOLOv8 model to obtain an object detection model, and evaluate its performance through the test set.

[0134] (3) By adjusting the hyperparameters of the model, screen out the object detection model with the best detection effect as the final version in the actual working scenario.

[0135] When the improved YOLOv8 model is applied, according to the working state of the lifting appliance, knowing the coordinate positions of the lifting appliance and the working container in the image, crop the approximate areas where the lifting appliance, the working container, and the connected container are located in the image as the detection area, and use the object detection model for detection to locate the positions of the lifting appliance, the working container, and the connected container.

[0136] S3: Analyze the anomaly detection area based on the pyramid hierarchical LK optical flow method to obtain the lifting height of the working container or the connected container.

[0137] In practical applications, the pyramid hierarchical LK optical flow method can be used to obtain the moving distance of the object in the image, that is, the lifting height of the working container or the connected container.

[0138] Specifically, step S3 includes:

[0139] S31: Perform grayscale processing and multiple downsamplings on the anomaly detection area to obtain a pyramid image sequence.

[0140] Perform grayscale processing on the anomaly detection area located by the object detection model in step S2; then apply a 5×5 Gaussian kernel with a standard deviation of to perform multiple downsamplings with a step size of 2, and each time the image is reduced to half of its original size. and are the grayscale values of a certain point on the image before and after sampling, respectively.

[0141]

[0142] Among them, the Gaussian kernel represents the Gaussian kernel similarity between the m-th vector and the n-th vector in the input space; l is the number of downsamplings; x and y are the pixel coordinate positions.

[0143] S32: Perform optical flow estimation from top to bottom on two consecutive sampled images.

[0144] Specifically, step S32 includes:

[0145] S321: Calculate the gray - level gradients of the image in the x - and y - axis directions and the gray - level gradient on the time axis at a certain point.

[0146] Starting from the top - most image and using two consecutive sampled images , calculate the image gradient.

[0147] Among them, respectively represent the gray - level gradients of the image in the direction along the axis, represents the gray - level gradient of on the time axis.

[0148] S322: According to the optical flow constraint equation, use the least - squares method to solve the optimal solution of the optical flow of two consecutive sampled images.

[0149] To solve the optical flow of a certain point in the image, use the 3×3 neighborhood window of this point as the constraint condition to solve the over - determined optical flow constraint equation . Among them, represents the gradient value of all points in the neighborhood window in the X - axis direction.

[0150] Since the number of equations is much larger than the number of unknowns, the least - squares method is needed to solve the optimal solution. Define the sum of squared residuals , where d is the optical flow.

[0151] Take the derivative of E with respect to d and set the derivative to zero, we get , then the solution is .

[0152] Use the displacement amount of the upper layer , to initialize the displacement amount of the lower - layer image:

[0153]

[0154] And correct the initialized displacement amount:

[0155]

[0156] S33: Use the optical flow estimation result to obtain the lifting height of the working container or the hooked container.

[0157] The moving distance of the target in the image is the optical flow value of the final original image, and the optical flow value of the final original image It is the segmented optical flow of all layers superimposed:

[0158]

[0159] Finally, according to the optical flow the lifting height of the working container or the connected container is obtained.

[0160] S4: According to the lifting height of the working container or the connected container, determine whether there is an abnormal connection of the container.

[0161] Specifically, step S4 includes: (1) When the spreader is in the unlocked state, if the lifting height of the working container exceeds the preset height threshold, it is determined that there is an abnormal connection.

[0162] According to the quay crane operation experience, a height threshold for the working container can be preset in advance. When the spreader is in the unlocked state, the working container should stay in place without lifting. Therefore, when it is detected that the working container has a certain lifting height exceeding the height threshold, it can be determined that there is an abnormal connection of the working container.

[0163] (2) When the spreader is in the locked state, if the lifting height of the connected container exceeds the preset height threshold, it is determined that there is an abnormal connection.

[0164] According to the quay crane operation experience, a height threshold for the connected container can be preset in advance. When the spreader is in the locked state, the working container rises with the spreader, while the connected container below should stay in place without lifting. Therefore, when it is detected that the connected container has a certain lifting height exceeding the height threshold, it can be determined that there is an abnormal connection of the connected container.

[0165] When it is determined that there is an abnormal connection, the server will send a stop signal to the PLC. After receiving it, the PLC will automatically pause the rising action of the spreader. At the same time, the server will send an alarm signal to notify the staff to handle the abnormal situation in time. If no abnormality is found and the spreader or the working container has reached the predetermined height, the operation will proceed normally.

[0166] Figure 4 The overall processing flow of this method is shown. In practical applications, the quay crane lifting image data can be collected in real time for detection to determine whether there is an abnormal connection, so as to facilitate timely processing.

[0167] The beneficial effect of this embodiment is that the target detection model in this method applies an improved YOLOv8 detection model. Adding the C2PFA module can improve the feature extraction ability of the model. Using multiple detection heads to detect the spreader, the working container and the connected container simultaneously can greatly reduce the computation amount and improve the detection accuracy.

[0168] Embodiment III

[0169] A ship-side container connection abnormality detection system based on machine vision, based on the ship-side container connection abnormality detection method based on machine vision described in Embodiment I or II, includes:

[0170] An acquisition module, configured to obtain the opening and closing state of the quay crane spreader and the monitoring video image at the quay crane spreader;

[0171] A detection module, configured to detect the monitoring video image at the quay crane spreader based on the target detection model according to the opening and closing state of the quay crane spreader, and obtain the abnormal detection area;

[0172] An optical flow analysis module, configured to analyze the abnormal detection area based on the pyramid hierarchical LK optical flow method to obtain the lifting height of the working container or the connected container;

[0173] A judgment module, configured to judge whether the container has an abnormal connection according to the lifting height of the working container or the connected container.

[0174] Embodiment IV

[0175] A computer program product includes a computer program, and when the computer program is executed, it implements the steps of the ship-side container connection abnormality detection method based on machine vision described in Embodiment I or II.

[0176] Embodiment V

[0177] A readable storage medium stores the computer program described in Embodiment IV, and when the computer program is executed, it implements the steps of the ship-side container connection abnormality detection method based on machine vision described in Embodiment I or II.

[0178] As mentioned above, it is only the specific implementation manners of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for detecting abnormal connection of ship-side containers based on machine vision, characterized in that, It includes the following steps: Obtain the opening and closing state of the quay crane spreader and the monitoring video image at the quay crane spreader; According to the opening and closing state of the quay crane spreader, detect the monitoring video image at the quay crane spreader based on the target detection model to obtain the abnormal detection area; Analyze the abnormal detection area based on the pyramid hierarchical LK optical flow method to obtain the lifting height of the working container or the hooked container; Judge whether the container has a hooking abnormality according to the lifting height of the working container or the hooked container; Among them, the detecting the monitoring video at the quay crane spreader based on the target detection model according to the opening and closing state of the quay crane spreader to obtain the abnormal detection area includes: When the spreader is in the unlocked state, the abnormal detection area is the spreader and the working container below the spreader; When the spreader is in the locked state, the abnormal detection area is the working container below the spreader and the hooked container below the working container; Among them, the analyzing the abnormal detection area based on the pyramid hierarchical LK optical flow method to obtain the lifting height of the working container or the hooked container includes: Perform grayscale processing and multiple downsamplings on the abnormal detection area to obtain a pyramid image sequence; Perform optical flow estimation from top to bottom on two consecutive sampled images; Use the optical flow estimation result to obtain the lifting height of the working container or the hooked container; Among them, the judging whether the container has a hooking abnormality according to the lifting height of the working container or the hooked container includes: When the spreader is in the unlocked state, if the lifting height of the working container exceeds the preset height threshold, it is judged that a hooking abnormality has occurred; When the spreader is in the locked state, if the lifting height of the hooked container exceeds the preset height threshold, it is judged that a hooking abnormality has occurred.

2. The method for detecting abnormal connection of ship-side containers based on machine vision according to claim 1, wherein The obtaining the opening and closing state of the quay crane spreader and the monitoring video image at the quay crane spreader includes: [[ID=!16]]The PLC of the quay crane controls the spreader to unlock or lock; Obtain the control signal of the PLC and judge the opening and closing state of the quay crane spreader; Use the vision device to collect the monitoring video image at the quay crane spreader.

3. The method for detecting abnormal connection of ship-side containers based on machine vision according to claim 1, wherein The detecting the monitoring video image at the quay crane spreader based on the target detection model includes: Use the YOLO, YOLOv8 or improved YOLOv8 target detection model to detect the monitoring video image at the quay crane spreader and identify the spreader, the working container and the hooked container.

4. The method for detecting abnormal connection of ship-side containers based on machine vision according to claim 3, wherein The detecting the monitoring video image at the quay crane spreader based on the target detection model includes: Build a YOLOv8 target detection model to detect and locate the spreader, the working container and the hooked container.

5. The method for detecting abnormal connection of ship-side containers based on machine vision according to claim 3, wherein The improved YOLOv8 model includes: The Backbone layer extracts the features of the monitoring video image based on the convolutional module, C2f module, SPPF module and C2PFA self-attention module; The Neck layer uses the features extracted by the Backbone layer to perform multi-scale feature fusion based on upsampling, feature splicing, convolutional module, C2f module, and FPN+PAN structure to obtain the fused features; The Head layer, according to the fused features, uses multiple detection heads to detect and locate the areas where the spreader, the working container and the hooked container are located.

6. The method for detecting abnormal connection of ship-side containers based on machine vision according to claim 1, characterized in that, The performing optical flow estimation from top to bottom on two consecutive sampled images includes: Calculate the gray - scale gradient of the image along the x - and y - axes and the gray - scale gradient on the time axis at a certain point; According to the optical flow constraint equation, use the least - squares method to solve the optimal solution of the optical flow of two consecutive sampled images.

7. A ship-side container connection anomaly detection system based on machine vision, characterized in that, The machine - vision - based abnormal detection method for container connection on the ship side according to any one of claims 1 to 6, includes: An acquisition module, configured to obtain the opening and closing state of the quay crane spreader and the monitoring video image at the quay crane spreader; A detection module, configured to detect the monitoring video image at the quay crane spreader based on the opening and closing state of the quay crane spreader and the target detection model, and obtain the abnormal detection area; An optical flow analysis module, configured to analyze the abnormal detection area based on the pyramid - hierarchical Lucas - Kanade optical flow method to obtain the lifting height of the working container or the connected container; A judgment module, configured to judge whether the container has an abnormal connection according to the lifting height of the working container or the connected container; Wherein, the detecting the monitoring video at the quay crane spreader based on the opening and closing state of the quay crane spreader and the target detection model to obtain the abnormal detection area includes: When the spreader is in the unlocked state, the abnormal detection area is the spreader and the working container below the spreader; When the spreader is in the locked state, the abnormal detection area is the working container below the spreader and the connected container below the working container; Wherein, the analyzing the abnormal detection area based on the pyramid - hierarchical Lucas - Kanade optical flow method to obtain the lifting height of the working container or the connected container includes: Perform gray - scale processing and multiple down - samplings on the abnormal detection area to obtain a pyramid image sequence; Perform optical flow estimation from top to bottom on two consecutive sampled images; Use the optical flow estimation result to obtain the lifting height of the working container or the connected container; Wherein, the judging whether the container has an abnormal connection according to the lifting height of the working container or the connected container includes: When the spreader is in the unlocked state, if the lifting height of the working container exceeds a preset height threshold, it is judged that an abnormal connection has occurred; When the spreader is in the locked state, if the lifting height of the connected container exceeds a preset height threshold, it is judged that an abnormal connection has occurred.

Images