Container bottom twist lock detection method, system and equipment and storage medium

Abstract

The invention provides a container bottom twist lock detection method, system and device and a storage medium, and belongs to the technical field of container twist lock detection, and the method comprises the steps: collecting a container bottom video stream, carrying out the preprocessing, and generating a preprocessed image; inputting the preprocessed images into a neural network model, performing twist lock detection on each image, outputting a target detection result, counting the number of images with targets in the target detection results of all the images, if the number is within a middle threshold range, determining that undetached twist locks possibly exist at the bottom of the current container, and if the number is not within a middle threshold range, determining that the undetached twist locks possibly exist at the bottom of the current container. And according to the target detection results of all the images, preliminarily associating the detected twist lock and judging whether the twist lock is stably identified, if so, determining that the twist lock which is not disassembled exists, and triggering an alarm mechanism, and otherwise, determining that the twist lock which is not disassembled does not exist. Whether the twist lock is arranged at the bottom of the container or not can be automatically detected in the lifting movement process of the lifting appliance, hovering inspection of the lifting appliance is not needed, and the probability that the twist lock at the bottom of the container is brought into a container area to be stacked is reduced.

Description

Technical Field

[0001] The present application relates to the technical field of container twist lock detection, and in particular to a method, system, device and storage medium for detecting a container bottom twist lock. Background Art

[0002] In the traditional port operation mode, after unloading the container from the cargo ship, the twist lock is manually removed by the staff at the quay crane twist lock station, and then the container is transported horizontally to the yard area by the container truck. However, this manual operation mode has obvious drawbacks:

[0003] Risk of missed twist locks: Due to the huge volume of container transportation and the complex operating procedures, workers may neglect to remove the twist locks on the bottom of the container, which may then be transported to the yard by the internal container truck and unloaded by the gantry crane to the container area for stacking. Safety hazards: If the twist locks on the bottom of the container are not removed and stacked directly, the bottom of the container will be unstable. This may not only cause the container on the current layer to overturn, but may also affect the stability of the upper container, triggering a chain reaction, causing damage to the container, damage to the cargo, and even endangering the lives of people. Limitations of manual confirmation: In a mode that relies entirely on the remote control driver to manually confirm whether the twist lock is on, the driver will easily become fatigued after working for a long time and it is difficult for him to always maintain a high level of concentration, further increasing the risk of missed detection.

[0004] To address these issues, some ports have attempted to use laser scanners. However, laser point cloud scanning cannot accurately identify foreign objects such as plastic bags on the bottom of containers, and can easily misidentify them as unremoved twist locks, leading to false alarms and impacting operational efficiency. Furthermore, laser inspection only provides point cloud data, not actual images of the container bottom. This hinders remote control drivers from performing secondary verification and makes it difficult to ensure the accuracy of inspection results.

[0005] In view of this, the present application proposes a method, system, device and storage medium for detecting twist locks at the bottom of a container, which can automatically detect twist locks that have not been removed from the bottom of the container, improve recognition efficiency, reduce the workload of remote control drivers, and avoid the risk of missed detection due to human negligence. Summary of the Invention

[0006] In order to solve the problems of high false detection probability and lack of intuitive image information in the existing technology, the present application provides a container bottom twist lock detection method, system, equipment and storage medium to solve the above technical defects.

[0007] In a first aspect, the present invention provides a method for detecting a twist lock at the bottom of a container, the method comprising the following steps:

[0008] S1. Capturing a video stream from the bottom of a container through a camera, and preprocessing the video stream to generate a preprocessed image.

[0009] S2. Input the preprocessed images into the neural network model, perform twist-lock detection on each image, and output the target detection results of each image, including: whether the target exists, and the center point coordinates and size of the target box when the target exists;

[0010] S3. Count the number of images containing the target in the target detection results of all images. If the number of images is within the middle threshold range, there may be an unremoved twist lock at the bottom of the current container. Based on the target detection results of all images, preliminarily associate the detected twist locks and determine whether they are stably identified. If so, there is an unremoved twist lock at the bottom of the container; otherwise, there is no unremoved twist lock.

[0011] S4. When there is an unremoved twist lock at the bottom of the container, the alarm mechanism is triggered and a control instruction is sent to the crane PLC control system to limit the action of the lifting mechanism.

[0012] Optionally, in step S3, the number of images containing targets in all image target detection results is counted. If the number is within an intermediate threshold range, then there may be an unremoved twist-lock at the bottom of the current container. The intermediate threshold range is between a set first threshold and a set second threshold. If the number of images containing targets is less than the set first threshold, then there is no unremoved twist-lock at the bottom of the current container. If the number of images containing targets is greater than the set second threshold, then there is an unremoved twist-lock at the bottom of the current container.

[0013] Optionally, in step S3, the detected twist-locks are preliminarily associated and judged whether they are stably identified based on the target detection results of all images. If so, there are unremoved twist-locks at the bottom of the container; otherwise, there are no unremoved twist-locks. Specifically, the following steps are included:

[0014] S31. For each camera, the image is divided into two areas, left and right. Based on the coordinates of the center point of the target frame of each image, it is determined whether there is a target in the left area and the right area of each image.

[0015] S32. If the target frame exists in the same area of the consecutive image frames, the two target frames in the area are associated as the same twist lock, and the operation of step S33 is executed; otherwise, the current frame image is filtered, and the operation of step S32 is executed for the next frame image;

[0016] S33, determining whether the target frame associated with the same twist lock satisfies the requirement that the width change rate is less than 5%. If so, proceed to step S34; otherwise, filter the current frame image and execute step S33 for the next frame image.

[0017] S34, determining whether the target frame associated with the same twist lock satisfies the requirement that the horizontal displacement of the center point is less than or equal to m1 pixels and the vertical displacement is greater than m2 pixels. If so, proceed to step S35; otherwise, filter the current frame image and execute step S34 for the next frame image.

[0018] S35. Determine whether all retained images satisfy the requirement that the target area standard deviation between N consecutive image frames is less than 8%. If so, it is determined that the twist-lock is stably recognized in the consecutive image frames of the current camera. Otherwise, the current camera does not have a stably recognized twist-lock.

[0019] S36. If at least one camera stably identifies a twist lock, then there is an unremoved twist lock at the bottom of the container; otherwise, there is no unremoved twist lock at the bottom of the container.

[0020] Optionally, in step S1, the two cameras are respectively installed on the left and right sides below the center of the trolley saddle beam, and the cameras collect the video stream of the bottom of the container after the spreader grabs the container and lifts it, and each camera collects the two corners of the container it faces.

[0021] Optionally, in step S1, preprocessing the bottom video stream to generate a preprocessed image includes:

[0022] S11, disassembling the collected container bottom video stream according to a preset frame rate to extract a continuous image frame sequence;

[0023] S12, adjusting the size of each extracted image frame to a uniform size, and dynamically enhancing the image contrast and texture features according to the ambient lighting or weather conditions to generate a preprocessed image.

[0024] Optionally, in step S2, the preprocessed image is input into a neural network model improved based on YOLOv8, wherein the neural network model improved based on YOLOv8 is a model in which a CBAM attention module is embedded in the backbone network and neck network of the YOLOv8 model, and based on a meta-learning framework, the weights of the classification loss and the positioning loss are dynamically adjusted according to the environmental interference level.

[0025] Optionally, step S4 specifically includes:

[0026] If there is an unremoved twist lock at the bottom of the container, a control instruction to limit the action of the lifting mechanism is sent to the crane PLC control system, and at the same time, an audible and visual alarm is triggered in the driver's cab and a pop-up prompt on the operation screen is triggered; the pop-up prompt on the operation screen includes: the lifting height of the spreader and the area where the twist lock has not been removed.

[0027] In a second aspect, a container bottom twist lock detection system is provided, the system comprising:

[0028] A camera, a switch, a PLC control system, and an AI controller, wherein the camera is connected to the AI controller via the switch, and the PLC control system is also connected to the AI controller; the AI controller includes a preprocessing module, a target detection module, a recognition module, and an early warning module;

[0029] The camera is configured to collect a video stream from the bottom of the container;

[0030] The preprocessing module is used to preprocess the bottom video stream to generate a preprocessed image;

[0031] The target detection module is used to input the preprocessed images into the neural network model, perform twist lock detection on each image, and output the target detection results of each image, including: whether the target exists, and the center point coordinates and size of the target box when the target exists;

[0032] The recognition module is configured to count the number of images containing the target in all the image target detection results. If the number is within an intermediate threshold range, it is possible that there is an unremoved twist lock at the bottom of the current container. Based on the target detection results of all the images, the detected twist locks are preliminarily associated and judged whether they are stably recognized. If so, there is an unremoved twist lock at the bottom of the container; otherwise, there is no unremoved twist lock.

[0033] The early warning module is used to trigger an alarm mechanism and send a control instruction to the crane PLC control system to limit the movement of the lifting mechanism when there is an unremoved twist lock at the bottom of the container.

[0034] The PLC control system is used to drive or limit the action of the lifting mechanism when receiving instructions.

[0035] In a third aspect, the present application proposes a terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above-mentioned container bottom twist lock detection methods when executing the computer program.

[0036] In a fourth aspect, the present application proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned methods for detecting a bottom twist lock of a container.

[0037] Compared with the prior art, the present invention has the following advantages:

[0038] (1) Improve detection accuracy and reliability: The neural network model can accurately identify suspicious targets with unremoved twist locks at the bottom of the container. By using preliminary association and continuous stable recognition, only suspicious targets that meet the conditions are judged as unremoved, effectively reducing the false alarm rate and ensuring the reliability of the detection results.

[0039] (2) Improve detection speed and reduce detection difficulty: Set thresholds such as the target frame width change rate, target center point displacement, and target area standard deviation, strictly screen and filter the detection results, and set the initial association method and continuous and stable recognition conditions according to the twist lock position of the container. Compared with traditional target tracking methods, it can greatly reduce the amount of calculation, improve detection speed, and reduce detection difficulty.

[0040] (3) Dynamic non-hovering detection: The present invention can autonomously detect whether there is a twist lock on the bottom of the box during the lifting movement of the sling, without the need for the sling to hover for inspection. Compared with the laser solution that requires the sling to hover and wait for laser scanning, the present invention can perform detection without interrupting the operation process, thereby improving detection efficiency.

[0041] (4) Enhance system adaptability and stability: Pre-process the bottom video stream, including image decomposition, size adjustment, and dynamic enhancement of image contrast and texture features according to ambient lighting or weather conditions, so that the system can work stably in different environments and adapt to the changing operating conditions of the port.

[0042] (5) Providing intuitive image information: Different from the laser scanning solution, the present invention uses a camera for target detection, which can provide real image information of the bottom of the container, making it easier for the remote control driver to make a second confirmation and avoid misjudgment.

[0043] (6) System integration and collaborative work: Integrate sensors such as cameras, PLC systems, and AI controllers into an organic whole. Through collaborative work between systems, automatic detection, real-time feedback, and effective control of the unremoved twist locks at the bottom of the container can be achieved, thereby improving the intelligence level and safety of the entire operation system.

[0044] (7) Improve operational efficiency and safety: Automatically detect unremoved twist locks on the bottom of containers, reduce the probability of the twist locks on the bottom of containers being brought into the stacking area, and reduce safety accidents such as container overturning, damage, and casualties caused by unremoved twist locks. At the same time, it reduces the workload of remote control drivers and avoids the risk of missed detection due to human negligence.

[0045] (8) High degree of intelligence and automation: The improved neural network model based on YOLOv8 is embedded in the CBAM attention module, and the loss weight is dynamically adjusted in combination with the meta-learning framework, so that the model can better adapt to the detection needs in complex environments and different conditions, and improve the detection efficiency and intelligence level. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Other features, objects and advantages of the present application will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0047] Figure 1 is a flow chart of a container bottom twist lock detection method according to the present application;

[0048] Figure 2 is a network topology diagram of a container bottom twist lock detection system according to the present application;

[0049] Figure 3 is a schematic diagram of the installation position of the camera according to the present application;

[0050] Figure 4 This is a detection effect diagram of the container bottom twist lock detection system according to the present application;

[0051] Figure 5 is another detection effect diagram of the container bottom twist lock detection system according to the present application;

[0052] Figure 6 This is another detection effect diagram of the container bottom twist lock detection system according to the present application;

[0053] Figure 7 It is a structural diagram of a computer system suitable for implementing the electronic device of the embodiment of the present application. DETAILED DESCRIPTION

[0054] The present application will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to explain the relevant inventions and are not intended to limit the inventions. It should also be noted that, for ease of description, only portions relevant to the relevant inventions are shown in the accompanying drawings. It should be noted that, unless there is a conflict, the embodiments and features in the embodiments of the present application may be combined with one another. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the examples.

[0055] Example 1

[0056] refer to Figure 1 As shown, the container bottom twist lock detection method proposed in this application includes the following steps:

[0057] S1. Capture the container bottom video stream through a camera, pre-process the bottom video stream, and generate a pre-processed image.

[0058] Figure 3 Shows a schematic diagram of the equipment installation location of this application, Figure 3 (d) is the front view of the saddle beam of the cart. Figure 3(a)-(c) are schematic cross-sectional views along directions A, B, and C in the front view, respectively; Figure 3 As shown in the figure, two cameras CA-01 and CA-02 are installed on the operating lane side of the electrical room, on the left and right sides below the center of the truck saddle beam. When the tire crane lifts the container from the container truck, the two cameras installed on the operating lane side of the gantry crane are used to obtain the bottom video stream of the container after the spreader grabs and lifts it within the camera's field of view. The untwisted lock algorithm of the AI controller deployed in the electrical room requests the front-end camera to obtain the video stream. That is, the container has four corners, and each camera captures the two corners of the container it faces.

[0059] In this embodiment, preprocessing the bottom video stream to generate a preprocessed image includes the following sub-steps:

[0060] S11. The collected container bottom video stream is decomposed at a preset frame rate (e.g., 30 frames per second) to extract a continuous sequence of image frames. Decomposing at a frame rate of 30 frames per second ensures that 30 images are acquired per second, meeting real-time requirements while providing sufficient detail for subsequent image processing and analysis. In actual applications, the frame rate setting can be flexibly adjusted to achieve optimal detection results based on the specific operating environment and detection requirements.

[0061] S12. Resize each extracted image frame to a uniform size (640px×640px), and dynamically enhance the image contrast and texture features according to ambient lighting or weather conditions to generate a preprocessed image.

[0062] Specifically, the camera's built-in light sensor or an external light sensor can be used to obtain real-time ambient light intensity data. Based on the actual application scenario, a baseline light value is pre-set. This baseline value is typically selected to achieve optimal image quality under normal operating lighting conditions. Based on the acquired current light value and the preset baseline light value, a correction value is calculated using the following formula. This correction value is used to guide subsequent adjustments to image preprocessing parameters.

[0063]

[0064] If the correction value is less than 1, it means the ambient light is low and the image brightness needs to be increased. You can use the formula "Adjusted brightness = original brightness × correction value" to increase the image brightness.

[0065] Similarly, for contrast, if the correction value is less than 1, the image contrast can be enhanced by "adjusted contrast = original contrast × correction value" to make the object boundaries and details in the image clearer.

[0066] In low-light conditions, the texture features of an image may not be clearly visible. Image enhancement algorithms, such as histogram equalization and filtering, can be used to further highlight texture details in the image and improve the recognizability of objects like twistlocks. Besides lighting factors, appropriate image processing strategies can also be adopted for different weather conditions, such as fog and rain. For example, in foggy conditions, dehazing algorithms can be used to reduce the impact of fog on image clarity; in rainy conditions, techniques such as image sharpening can be used to enhance image clarity and detail.

[0067] Continue to refer Figure 1 The container bottom twist lock detection method proposed in this application also includes the following steps:

[0068] S2. Input the preprocessed images into the neural network model, perform twist lock detection on each image, and output the target detection results of each image, including: whether there is a target, and the center point coordinates and size of the target box when the target exists.

[0069] In this embodiment, the preprocessed image can be input into a neural network model improved based on YOLOv8, wherein the backbone network (Backbone) and neck network of the improved YOLOv8 neural network model are embedded with the CBAM (Convolutional Block Attention Module) attention module. CBAM includes a channel attention module and a spatial attention module, which can dynamically focus on key areas in the image. The channel attention module obtains the important features of the channel through global average pooling and maximum pooling operations, and obtains the channel attention weight after processing through a fully connected layer and an activation function. The spatial attention module further refines the feature map processed by the channel attention, capturing the important areas of the spatial dimension in the feature map through convolution operations. This multi-scale attention mechanism enables the model to more accurately focus on key target areas such as twist locks, thereby improving detection accuracy. First, the channel weights of the feature map are adjusted through the channel attention mechanism to obtain a weighted feature map. Then, the spatial weights of the feature map are adjusted through the spatial attention mechanism, and the feature map is weighted again. The final output feature map includes an enhanced representation of the channel and spatial dimensions.

[0070] To enable the model to better adapt to different environmental conditions, a meta-learning framework was introduced. This meta-learning framework allows the model to learn how to quickly adapt to new environments and tasks during training. Specifically, through dynamic loss function weight adjustment, the model can adjust the weights of classification loss and localization loss in real time based on the level of environmental interference (such as light intensity and weather conditions). For example, in rainy and foggy weather, the model will increase the weight of texture features to enhance the recognition of twistlock targets. This adaptive detection algorithm enables the model to maintain high performance in complex and changing environments.

[0071] The target detection results for each image include whether there is a target, the center point coordinates and size of the target box when the target is present. That is, whether there is a target can be represented by "0" or "1", and the size of the target box includes width and height.

[0072] Continue to refer Figure 1 The container bottom twist lock detection method proposed in this application also includes the following steps:

[0073] S3. Count the number of images containing the target in the target detection results of all images. If the number of images is within the middle threshold range, there may be an unremoved twist lock at the bottom of the current container. Based on the target detection results of all images, preliminarily associate the detected twist locks and determine whether they are stably identified. If so, there is an unremoved twist lock at the bottom of the container. Otherwise, there is no unremoved twist lock.

[0074] The intermediate threshold range is between the set first threshold and the second threshold. If the number of images with the target is less than the set first threshold, there is no unremoved twist lock at the bottom of the current container. If the number of images with the target is greater than the set second threshold, there is an unremoved twist lock at the bottom of the current container. The first threshold and the second threshold can be determined based on expert experience.

[0075] Of course, in order to further improve the accuracy of determining whether there may be unremoved twist-locks, an intermediate threshold range ([n1, n2]) can be set for the images captured by a single camera. That is, the number of images with targets in the target detection results of all images of each camera is counted separately. If the number of images with targets in one of the cameras is within [n1, n2], then there may be an unremoved twist-lock at the bottom of the current container.

[0076] Based on the target detection results of all images, the detected twist locks are preliminarily associated and judged whether they are stably identified. If so, there are unremoved twist locks at the bottom of the container. Otherwise, there are no unremoved twist locks. The specific steps include:

[0077] S31 , for each camera, dividing the image into two areas, left and right, and judging whether there is a target in the left area and the right area of each image according to the center coordinates of the target frame of each image.

[0078] Since the container only has twist locks at the four corners and the container always remains horizontally raised and lowered without twisting during the lifting process, dividing the image equally into left and right parts can make the target association faster, simpler and more rapid.

[0079] S32. If there are target frames in the same area of the previous and next consecutive image frames, the two target frames in the area are associated as the same twist lock and the operation of step S33 is performed; otherwise, the current frame image is filtered and the operation of step S32 is performed on the next frame image.

[0080] S33. Determine whether the target frame associated with the same twist lock satisfies the requirement that the width change rate is less than 5%. If so, continue to perform the operation of step S34. Otherwise, filter the current frame image and perform the operation of step S33 on the next frame image.

[0081] Specifically, for the twist-lock target in consecutive image frames, the rate of change of its target frame width is calculated. The specific formula is:

[0082] Where W t Indicates the current frame target box width, W t-1 Indicates the width of the target box in the previous frame.

[0083] This threshold is set to eliminate size mutations caused by factors such as viewing angle changes or image noise, ensuring that only targets with smaller size changes enter subsequent judgment.

[0084] S34. Determine whether the target frame associated with the same twist lock satisfies the requirement that the horizontal displacement of the center point is less than or equal to m1 pixels and the vertical displacement is greater than m2 pixels. If so, continue with step S35. Otherwise, filter the current frame image and execute step S34 for the next frame image.

[0085] Specifically, for the twist-lock target in consecutive image frames, the horizontal and vertical displacements of its center point are calculated. Horizontal displacement = X-coordinate of the target center point in the current frame - X-coordinate of the target center point in the previous frame; vertical displacement = Y-coordinate of the target center point in the current frame - Y-coordinate of the target center point in the previous frame.

[0086] If the horizontal displacement is less than or equal to 10 pixels and the vertical displacement is greater than 100 pixels, the twistlock target is considered to be in the same position as if it were still in place, and the process continues to step S35. Otherwise, the current frame is filtered, and step S34 is performed on the next frame. The horizontal displacement threshold is set to eliminate lateral deviations caused by camera shake or slight target movement, while the vertical displacement threshold is set based on the expected trajectory of the twistlock during the spreader's ascent.

[0087] S35. Determine whether all retained images satisfy the requirement that the target area standard deviation between N consecutive image frames is less than 8%. If so, it is determined that the twist-lock is stably recognized in the consecutive image frames of the current camera. Otherwise, the current camera does not have a stably recognized twist-lock.

[0088] Specifically, for the twist-lock target in N consecutive image frames, the standard deviation of the target area is calculated as follows:

[0089]

[0090] Where A i Indicates the target area per frame; represents the average target area; N represents the number of frames.

[0091] The 8% threshold is set to ensure that the area of the twist-lock target in consecutive frames changes little, and the recognition process is stable and reliable, thereby improving the accuracy of the judgment.

[0092] S36. If at least one camera stably identifies a twist lock, then there is an unremoved twist lock at the bottom of the container; otherwise, there is no unremoved twist lock at the bottom of the container.

[0093] In a specific embodiment, if a twist lock target meets the following conditions in 5 consecutive frames: center point movement distance: 8 pixels (horizontally) + 105 pixels (vertically), width change rate is 3%, and area standard deviation is 6%, it is determined to be a "stable moving target" and a twist lock not removed alarm is triggered.

[0094] Continue to refer Figure 1 The container bottom twist lock detection method proposed in this application also includes the following steps:

[0095] S4. When there is an unremoved twist lock at the bottom of the container, the alarm mechanism is triggered and a control instruction is sent to the crane PLC control system to limit the action of the lifting mechanism.

[0096] If there is an unremoved twist lock at the bottom of the container, a control instruction to limit the action of the lifting mechanism is sent to the crane PLC control system, and at the same time, an audible and visual alarm is triggered in the driver's cab and a pop-up prompt on the operation screen is triggered; the pop-up prompt on the operation screen includes: the lifting height of the spreader and the area where the twist lock has not been removed.

[0097] The TCP protocol can be used to identify the real-time lifting height of the spreader and the area where the twist lock has not been removed. For example, if the camera on the left side recognizes and detects the existence of an unremoved twist lock, if the unremoved twist lock is on the left side of the captured image, then the unremoved twist lock is located at the left front corner of the container. Similarly, if the camera on the right side recognizes and detects the existence of an unremoved twist lock, if the unremoved twist lock is on the left side of the captured image, then the unremoved twist lock is located at the right rear corner of the container.

[0098] Example 2

[0099] refer to Figure 2 , the present application also proposes a container bottom twist lock detection system, the system comprising:

[0100] Camera, switch, PLC control system and AI controller. The camera is connected to the AI controller through the switch, and the PLC control system is also connected to the AI controller. The AI controller includes a preprocessing module, a target detection module, a recognition module and an early warning module.

[0101] The camera is configured to collect the video stream from the bottom of the container;

[0102] A preprocessing module, configured to preprocess the bottom video stream to generate a preprocessed image;

[0103] The target detection module is used to input the preprocessed images into the neural network model, perform twist lock detection on each image, and output the target detection results of each image, including: whether the target exists, and the center point coordinates and size of the target box when the target exists;

[0104] The recognition module is used to count the number of images containing the target in all image object detection results. If the number is within the middle threshold range, there may be an unremoved twist lock at the bottom of the current container. Based on the object detection results of all images, the detected twist locks are preliminarily associated and judged whether they are stably recognized. If so, there is an unremoved twist lock at the bottom of the container; otherwise, there is no unremoved twist lock.

[0105] The early warning module is used to trigger the alarm mechanism and send control instructions to the crane PLC control system to limit the movement of the lifting mechanism when there is an unremoved twist lock at the bottom of the container.

[0106] The PLC control system is used to drive or limit the action of the lifting mechanism when receiving instructions.

[0107] For more specific details about the above method, please refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be described again here.

[0108] During the container lifting process, the container bottom twist lock detection system provided by this application can automatically detect whether the four corner twist locks at the bottom of 20-foot, 40-foot, 45-foot, and double 20-foot standard containers have been removed, which is determined by the camera layout position and camera clarity.

[0109] After the AI visual dynamic non-hovering recognition container bottom twist lock detection system is started, it automatically obtains the real-time status information of the equipment from the crane PLC control system and feeds back the current system status to the crane PLC control system.

[0110] When the system is turned on and the equipment status and the container bottom twist lock detection system status are normal, after the container truck arrives at the working position, when the trolley enters above the working lane, and the spreader correctly completes the locking action, when the lifting mechanism rises, the camera of the detection system will identify and detect in real time whether there is a twist lock at the bottom of the container. If there is a twist lock at the bottom of the container that has not been removed, the detection system will feed back the result information of the twist lock detection to the crane PLC control system for the lifting mechanism rise limit strategy to limit the lifting mechanism action.

[0111] Preferably, the camera is of Hikvision / Uniview / Dahua brand, the switch is of Sanwang / Risecom / TP-LINK brand, the AI controller is of HIRI / EP series, and the unremoved twist lock detection software is of HIRI or customized.

[0112] Specifically, the left CA-01 camera is mounted on the working lane, below the center of the trolley saddle beam to the left, with a 300mm channel steel reserved for bracing. The camera's three-dimensional base is welded to the channel steel. The left camera is mounted on the base, connected to the 12V switch in the electrical room for power. A Category 6a network cable is then connected to the network port of the video switch in the electrical room to communicate with the AI controller, which in turn communicates with the PLC.

[0113] The right camera, a CA-02, is mounted on the working track, just to the right of the center of the trolley saddle beam. A 300mm channel steel is reserved, eliminating the need for diagonal bracing. The right camera's three-dimensional base is welded to the welded channel steel. The camera is mounted on the base and connected to the 12V power supply in the electrical room. A Category 6a network cable is then connected to the network port of the video switch in the electrical room to communicate with the AI controller, which in turn communicates with the PLC.

[0114] The switch is installed in the electrical room on a DIN-rail. The AI controller is installed in the cabinet of the intelligent system electrical room.

[0115] The camera power cables connect to a 12V switching power supply in the electrical room's distribution box, while the AI controller draws its AC 220V power from a nearby device. The network cables for the two cameras on the work lane side connect to a switch in the electrical room's cabinet, and the AI controller's network cable connects to the switch in the electrical room's cabinet for communication.

[0116] like Figure 4-Figure 6 As shown, Figure 4 (a)-(c) show the detection effect of accurately identifying the unremoved twist lock under three different shadows and high contrast conditions; Figure 5 (a)-(f) show the detection effect diagrams of accurately identifying the unremoved twist lock under different strong light conditions. Figure 6 (a) and (b) in FIG. 1 respectively represent the detection effect diagrams of the filtered image frames when making the final judgment on two possible twist locks that have not been disassembled; Figure 4-Figure 6As shown, the system of the present application can still accurately identify undisassembled twist locks under complex lighting conditions such as shadows and high contrast. This strong environmental adaptability ensures the stability and reliability of the system in actual port operations.

[0117] This application's container bottom twistlock detection system utilizes advanced AI vision technology and multimodal sensor fusion techniques. Through a neural network model embedded in a CBAM attention module and dynamically adjusted loss function weights, it achieves efficient detection of container bottom twistlocks. The system not only processes video streams captured by high-definition cameras in real time but also further improves detection accuracy through multi-frame image stability analysis and motion trajectory modeling.

[0118] In actual inspection operations, the system consistently achieves an accuracy rate of 97% or higher. This high accuracy is achieved by the system's precise recording and analysis of the twistlock's size, position changes, and movement distance across multiple image frames. Through strict threshold settings and multi-condition screening, the system effectively filters out false detections and missed detections, ensuring that only genuine, intact twistlocks are identified and trigger an alarm.

[0119] Furthermore, the system features an adaptive detection algorithm and incremental learning mechanism that dynamically adjusts model parameters based on environmental interference levels and automatically labels and retrains false positives and missed positives. This continuous optimization capability enables the system to continuously adapt to new detection environments and conditions, maintaining optimal detection performance.

[0120] In summary, the container bottom twist lock detection system of this application provides a solid technical guarantee for the safe and efficient operation of the port industry through its excellent detection accuracy, strong environmental adaptability and efficient real-time feedback.

[0121] Example 3

[0122] The present application also proposes a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above-mentioned container bottom twist lock detection methods when executing the computer program.

[0123] For more specific details about the above method, please refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be described again here.

[0124] Example 4

[0125] The present application also proposes a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the steps of any of the above-mentioned methods for detecting a bottom twist lock of a container are implemented.

[0126] Reference below Figure 7 , which shows a structural diagram of a computer system 500 suitable for implementing an electronic device of an embodiment of the present application. Figure 7 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0127] like Figure 7 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage unit 508 into a random access memory (RAM) 503. Various programs and data required for the operation of the computer system 500 are also stored in the RAM 503. The CPU 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0128] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, and the like; an output section 507 including a liquid crystal display (LCD) and speakers; a storage section 508 including a hard disk; and a communication section 509 including a network interface card such as a LAN card or a modem. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 510 as needed, so that computer programs read therefrom can be installed into the storage section 508 as needed.

[0129] For more specific details about the above method, please refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be described again here.

[0130] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 509, and / or installed from a removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above-mentioned functions defined in the method of the present application are performed.

[0131] It should be noted that the computer-readable storage medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this application, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code contained on a computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical cable, RF, or any suitable combination thereof.

[0132] Computer program code for performing the operations of the present application can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0133] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0134] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned inventive concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A method for detecting twist locks at the bottom of a container, characterized in that: The following steps are involved: S1. Capturing a video stream from the bottom of a container through a camera, and preprocessing the video stream to generate a preprocessed image. S2. Input the preprocessed images into the neural network model, perform twist lock detection on each image, and output the target detection results of each image, including: whether the target exists and the center point coordinates and size of the target box when the target exists; S3. Count the number of images containing the target in the target detection results of all images. If the number of images is within the middle threshold range, there may be an unremoved twist lock at the bottom of the current container. Based on the target detection results of all images, preliminarily associate the detected twist locks and determine whether they are stably identified. If so, there is an unremoved twist lock at the bottom of the container; otherwise, there is no unremoved twist lock. S4. When there is an unremoved twist lock at the bottom of the container, the alarm mechanism is triggered and a control instruction is sent to the crane PLC control system to limit the action of the lifting mechanism.

2. The method for detecting a twist lock on the bottom of a container according to claim 1, wherein: In step S3, the number of images containing targets in all the image target detection results is counted. If the number is within an intermediate threshold range, then there may be an unremoved twist-lock at the bottom of the current container. The intermediate threshold range is between a set first threshold and a set second threshold. If the number of images containing targets is less than the set first threshold, then there is no unremoved twist-lock at the bottom of the current container. If the number of images containing targets is greater than the set second threshold, then there is an unremoved twist-lock at the bottom of the current container.

3. The method for detecting container bottom twist locks according to claim 1, wherein: In step S3, based on the target detection results of all images, the detected twist locks are preliminarily associated and judged whether they are stably identified. If so, there are unremoved twist locks at the bottom of the container; otherwise, there are no unremoved twist locks. Specifically, the following steps are included: S31. For each camera, the image is divided into two areas, left and right. Based on the coordinates of the center point of the target frame of each image, it is determined whether there is a target in the left area and the right area of each image. S32. If the target frame exists in the same area of the consecutive image frames, the two target frames in the area are associated as the same twist lock, and the operation of step S33 is executed; otherwise, the current frame image is filtered, and the operation of step S32 is executed for the next frame image; S33, determining whether the target frame associated with the same twist lock satisfies the requirement that the width change rate is less than 5%. If so, proceed to step S34; otherwise, filter the current frame image and execute step S33 for the next frame image. S34, determining whether the target frame associated with the same twist lock satisfies the requirement that the horizontal displacement of the center point is less than or equal to m1 pixels and the vertical displacement is greater than m2 pixels. If so, proceed to step S35; otherwise, filter the current frame image and execute step S34 for the next frame image. S35. Determine whether all retained images satisfy the requirement that the target area standard deviation between N consecutive image frames is less than 8%. If so, it is determined that the twist-lock is stably recognized in the consecutive image frames of the current camera. Otherwise, the current camera does not have a stably recognized twist-lock. S36. If at least one camera stably identifies a twist lock, then there is an unremoved twist lock at the bottom of the container; otherwise, there is no unremoved twist lock at the bottom of the container.

4. The method for detecting container bottom twist locks according to claim 1, wherein: In step S1, the two cameras are respectively installed on the left and right sides below the center of the trolley saddle beam, and the cameras collect the video stream of the bottom of the container after the spreader grabs the container and lifts it, and each camera collects the two corners of the container it faces.

5. The method for detecting container bottom twist locks according to claim 1, wherein: In step S1, the bottom video stream is preprocessed to generate a preprocessed image, including: S11, disassembling the collected container bottom video stream according to a preset frame rate to extract a continuous image frame sequence; S12, adjusting the size of each extracted image frame to a uniform size, and dynamically enhancing the image contrast and texture features according to the ambient lighting or weather conditions to generate a preprocessed image.

6. The method for detecting container bottom twist locks according to claim 1, wherein: In step S2, the preprocessed image is input into a neural network model improved based on YOLOv8, wherein the neural network model improved based on YOLOv8 is to embed a CBAM attention module in the backbone network and neck network of the YOLOv8 model, and based on a meta-learning framework, dynamically adjust the weights of the classification loss and the positioning loss according to the environmental interference level.

7. The method for detecting container bottom twist locks according to claim 1, wherein: Step S4 specifically includes: If there is an unremoved twist lock at the bottom of the container, a control instruction to limit the action of the lifting mechanism is sent to the crane PLC control system, and at the same time, an audible and visual alarm is triggered in the driver's cab and a pop-up prompt on the operation screen is triggered; the pop-up prompt on the operation screen includes: the lifting height of the spreader and the area where the twist lock has not been removed.

8. A container bottom twist lock detection system, characterized in that: The system comprises: A camera, a switch, a PLC control system, and an AI controller, wherein the camera is connected to the AI controller via the switch, and the PLC control system is also connected to the AI controller; the AI controller includes a preprocessing module, a target detection module, a recognition module, and an early warning module; The camera is configured to collect a video stream from the bottom of the container; The preprocessing module is used to preprocess the bottom video stream to generate a preprocessed image; The target detection module is used to input the preprocessed images into the neural network model, perform twist lock detection on each image, and output the target detection results of each image, including: whether the target exists, and the center point coordinates and size of the target box when the target exists; The recognition module is configured to count the number of images containing the target in all the image target detection results. If the number is within an intermediate threshold range, it is possible that there is an unremoved twist lock at the bottom of the current container. Based on the target detection results of all the images, the detected twist locks are preliminarily associated and judged whether they are stably recognized. If so, there is an unremoved twist lock at the bottom of the container; otherwise, there is no unremoved twist lock. The early warning module is used to trigger an alarm mechanism and send a control instruction to the crane PLC control system to limit the movement of the lifting mechanism when there is an unremoved twist lock at the bottom of the container. The PLC control system is used to drive or limit the action of the lifting mechanism when receiving instructions.

9. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the container bottom twist-lock detection method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the container bottom twist-lock detection method according to any one of claims 1 to 7 are implemented.

Images