Container lifting detection method and system based on detection positioning combined with KCF tracking

Abstract

The present application aims at the problems of poor real-time performance, easy interference of complex environment and high misjudgment rate in the existing loading and unloading operation of truck lifting anomaly detection, and provides a truck lifting detection method and system based on detection positioning and KCF tracking combination, belonging to the technical field of port intelligent monitoring and computer vision detection. The method comprises the following steps: real-time acquisition of port operation area video stream and extraction of key frame image; using a target detection model to locate the container hole as a first tracking area; calculating the truck pallet position as a second tracking area by a space correlation derivation model; continuously tracking the motion trajectory of the two areas by using KCF algorithm; three-dimensional judgment based on distance, angle and distance change rate to determine the separation state of truck and container; triggering alarm in abnormal situation. Through multi-dimensional feature judgment, the stability and accuracy of container separation state recognition in complex environment are improved, which is suitable for safety monitoring of automatic container loading and unloading scene in port, wharf and the like.

Description

Technical Field

[0001] This invention relates to the field of port intelligent monitoring and computer vision detection technology, and in particular to a truck lifting detection method and system based on a combination of target detection and positioning and KCF target tracking algorithm. It is applicable to port, dock, logistics hub and automated container loading and unloading operation scenarios to perform safety monitoring and abnormal alarm on the separation state of truck and container during the lifting process. Background Technology

[0002] In port container loading and unloading operations, the coordinated operation of container trucks (container transport vehicles) and quay cranes or yard cranes is the core link to achieve efficient container transfer. When the lifting equipment lifts the container, it is essential to ensure that the truck and the container are completely detached. If the truck is lifted along with the container due to the truck's locking mechanism not being released or abnormal operation, it will pose a serious threat to the safety of vehicles, lifting equipment, and personnel on site.

[0003] Currently, port safety monitoring mainly relies on manual observation or simple video analysis for judgment. These methods typically depend on single features or simple threshold judgments, which are prone to high false positive and false negative rates in complex port environments with drastic changes in lighting, equipment obstruction, mechanical vibration, and complex background interference. Furthermore, traditional video detection methods often require detecting and locating multiple target areas in the image separately, which is not only computationally intensive but also struggles to balance detection accuracy and real-time performance when processing high-resolution video streams. This makes it difficult to meet the stringent requirements of modern smart port automated operations for high stability and real-time performance in safety monitoring systems. Summary of the Invention

[0004] This invention addresses the problems in existing technologies where port container loading and unloading operations mainly rely on manual observation or simple video analysis, resulting in poor real-time performance, insufficient stability, susceptibility to interference from complex environments, and high rates of false positives and false negatives in truck lifting anomaly detection. It provides a truck lifting detection method and system based on a combination of detection positioning and KCF tracking.

[0005] The technical solution adopted in this invention is:

[0006] A truck lifting detection method based on the combination of detection and positioning and KCF tracking includes the following steps:

[0007] Step 1: Real-time acquisition of video streams of the port operation area, and extraction of key frame images from the video stream according to a preset extraction strategy; the key frame image is an image frame that completely covers at least one container opening and the core detection target of the truck pallet.

[0008] Step 2: Use a pre-trained target detection model to accurately locate the container openings in the keyframe image, and set the area where the located container openings are located as the first tracking area;

[0009] Step 3: Based on the position information of the first tracking area in the image coordinate system of the key frame image, the position of the truck pallet under the container is calculated using a preset spatial correlation derivation model, and the area where the truck pallet is located is set as the second tracking area;

[0010] Step 4: During the container lifting process, the first tracking area and the corresponding second tracking area are used as the initial target templates of the KCF tracker to track the first tracking area and the corresponding second tracking area in the continuous video frames of the video stream, and the motion trajectory of the first tracking area and the corresponding second tracking area is obtained in real time.

[0011] Step 5: Based on the motion trajectory of the first tracking area and the corresponding second tracking area, calculate the distance, angle and distance change rate between the first tracking area and the corresponding second tracking area, and perform three-dimensional determination; wherein, the distance is the perpendicular distance between the first tracking area and the corresponding second tracking area, the angle is the motion trajectory angle of the second tracking area, and the distance change rate is the rate of change of the distance between the first tracking area and the corresponding second tracking area over time.

[0012] Step 6: When the judgment result is abnormal lifting, the truck lifting detection system triggers an alarm signal.

[0013] Furthermore, in step 2, the pre-trained target detection model is used to accurately locate the container openings in the keyframe image, and the area where the located container openings are located is set as the first tracking area. The specific process includes:

[0014] Step 21: Set the target detection model to the YOLO detection model;

[0015] Step 22: Obtain image samples of port operation areas containing container openings, and label the locations of the container openings in the image samples to obtain a container opening training sample set;

[0016] Step 23: Train the YOLO detection model using the container hole training sample set to obtain a pre-trained YOLO detection model;

[0017] Step 24: Input the keyframe image into the pre-trained YOLO detection model, and the pre-trained YOLO detection model outputs the container hole detection box and its detection confidence, as well as the position information of the center point of the container hole detection box in the image coordinate system of the keyframe image;

[0018] Step 25: Sort the detection confidence scores from high to low, and take the container hole detection frames corresponding to the highest sorted detection confidence scores as valid container hole detection results, and determine the first tracking area based on the valid container hole detection results.

[0019] Further, in step 25, determining the first tracking area based on the effective container opening detection results specifically includes:

[0020] The effective container opening detection results are expressed as follows:

[0021] B h =(x h ,y h ,w h ,h h );

[0022] Among them, B h x represents the container hole detection bounding box output by the pre-trained YOLO detection model. h y h These represent the x and y coordinates of the center point of the container opening detection frame in the image coordinate system of the keyframe image, respectively. h h h These represent the width and height of the container opening detection frame, respectively.

[0023] The first tracking area is generated based on the container hole detection frame:

[0024]

[0025] in:

[0026] x1=x h ;

[0027] y1=y h ;

[0028] ;

[0029] ;

[0030] Where R1 represents the first tracking region, x1 and y1 represent the x and y coordinates of the center point of the first tracking region in the image coordinate system of the keyframe image, respectively, and w1 and h1 represent the width and height of the first tracking region, respectively; x h y h These represent the x and y coordinates of the center point of the container opening detection frame in the image coordinate system of the keyframe image, respectively; w h h h These represent the width and height of the container opening inspection frame, respectively. Indicates the horizontal expansion coefficient. This represents the vertical expansion coefficient, and , .

[0031] Further, in step 3, based on the position information of the first tracking area in the image coordinate system of the keyframe image, the position of the truck pallet below the container is calculated using a preset spatial correlation derivation model, and the area where the truck pallet is located is set as the second tracking area. The specific process includes:

[0032] Step 31: Obtain the coordinates of the center point, width, and height of the first tracking region in the image coordinate system;

[0033] Step 32: Call the preset spatial correlation derivation model, which takes the center point coordinates, width, height and preset spatial correlation parameters of the first tracking area as input;

[0034] Step 33: Output the center point coordinates, width, and height of the second tracking region from the spatial correlation derivation model;

[0035] Step 34: Determine the area where the truck tray is located in the keyframe image based on the center point coordinates, area width, and area height of the second tracking area.

[0036] Furthermore, in step 3, when the target detection model detects different numbers of container openings in the keyframe image, it determines the position for deriving the center point of the second tracking region in different ways. The specific process includes:

[0037] When the target detection model detects only one container opening in the keyframe image, the recognition process obtains the first tracking region of the container opening. Then, the spatial association derivation model derives the position of the second tracking region based on the first tracking region corresponding to the single container opening, specifically including:

[0038] Obtain the center point of the first tracking area corresponding to the container opening:

[0039] C1=(x1,y1);

[0040] Wherein, C1 represents the center point of the first tracking area, and x1 and y1 represent the horizontal and vertical coordinates of the center point of the first tracking area corresponding to the container opening in the image coordinate system of the keyframe image, respectively.

[0041] The center point of the second tracking area is:

[0042] ;

[0043] Where C2 represents the center point of the second tracking area, x1 and y1 represent the horizontal and vertical coordinates of the center point of the first tracking area corresponding to the container opening in the image coordinate system of the keyframe image, respectively, kx represents the preset horizontal offset parameter, and ky represents the preset vertical offset parameter;

[0044] When the target detection model simultaneously detects multiple container openings on the same side in the keyframe image, it selects the two container openings with the highest detection confidence scores and performs recognition processing on the left and right container openings in the keyframe image to obtain the first tracking region corresponding to the left container opening and the first tracking region corresponding to the right container opening. Then, the spatial association derivation model deduces the position of the corresponding second tracking region based on the first tracking regions corresponding to the two container openings, specifically including:

[0045] Obtain the center point of the first tracking area corresponding to the opening of the left container:

[0046] C 1L =(x 1L ,y 1L );

[0047] Among them, C 1L x represents the center point of the first tracking area corresponding to the opening of the left container. 1L y 1L These represent the x and y coordinates of the center point of the first tracking area corresponding to the left container opening in the image coordinate system of the keyframe image, respectively.

[0048] And the center point of the first tracking area corresponding to the right container opening:

[0049] C 1R =(x 1R ,y 1R );

[0050] Among them, C 1R x represents the center point of the first tracking area corresponding to the right container opening. 1R y1R These represent the x and y coordinates of the center point of the first tracking area corresponding to the container opening on the right in the image coordinate system of the keyframe image, respectively.

[0051] The reference position of the lower edge of the container is determined based on two center points, and the center point of the second tracking area is determined according to the preset horizontal offset parameter kx and the preset vertical offset parameter ky.

[0052] ;

[0053] Where C2 represents the center point of the second tracking region, x 1L y 1L These represent the x and y coordinates of the center point of the first tracking area corresponding to the opening of the left container in the image coordinate system of the keyframe image, respectively. 1R y 1R These represent the x and y coordinates of the center point of the first tracking area corresponding to the right container opening in the image coordinate system of the keyframe image, respectively. kx represents the preset horizontal offset parameter, and ky represents the preset vertical offset parameter.

[0054] Furthermore, the spatial correlation derivation model is constructed in the following manner:

[0055] Multiple calibration sample images are collected from the same camera viewpoint within the port operation area. The calibration sample images include container openings and corresponding truck pallets, and the container opening area and the corresponding truck pallet area are marked in the calibration sample images respectively.

[0056] Based on the center point coordinates, width, and height of the container opening area, and the center point coordinates, width, and height of the truck pallet area, calculate the spatial correlation parameters of the truck pallet area relative to the container opening area. The spatial correlation parameters include at least horizontal offset parameters, vertical offset parameters, width ratio parameters, and height ratio parameters.

[0057] Statistical processing is performed on the spatial correlation parameters corresponding to multiple frames of the calibration sample images to obtain preset spatial correlation parameters; the preset spatial correlation parameters are configured into the initial spatial correlation derivation model to obtain the spatial correlation derivation model, so that the spatial correlation derivation model can output the center point coordinates, region width, and region height of the second tracking region corresponding to the first tracking region based on the input center point coordinates, region width, and region height of the first tracking region.

[0058] Further, in step 4, KCF tracking is performed on the first tracking region and the corresponding second tracking region. The specific process includes:

[0059] Step 41: Set the first tracking region as the initial target template of the first KCF tracker, and the second tracking region as the initial target template of the second KCF tracker;

[0060] Step 42: Calculate the correlation response maps of the first KCF tracker and the second KCF tracker in consecutive video frames of the video stream, respectively;

[0061] Step 43: Determine the positions corresponding to the pixels with the largest response values ​​in the relevant response map as the positions of the first tracking region and the corresponding second tracking region in the current frame, respectively;

[0062] Step 44: Update the target template of the corresponding KCF tracker according to the position of the first tracking region and the corresponding position of the second tracking region in the current frame;

[0063] Step 45: When the maximum response value of the correlation response map of the two KCF trackers is greater than or equal to the preset response threshold, the position corresponding to the pixel with the largest response value in the correlation response map is determined as the first tracking region position and the corresponding second tracking region position in the current frame, respectively; when the maximum response value of the correlation response map of any KCF tracker is lower than the preset response threshold, the target detection model is called again to locate the container opening, and the first tracking region and the corresponding second tracking region are re-initialized through the spatial association derivation model.

[0064] Further, in step 4, the motion trajectory includes the position sequence of the first tracking region in consecutive video frames of the video stream and the position sequence of the second tracking region in consecutive video frames of the video stream;

[0065] Let the position of the first tracking region in frame t be:

[0066]

[0067] Where t represents the video frame number. This represents the first tracking region in frame t. Let x and y represent the x and y coordinates of the center point of the first tracking region in the image coordinate system in frame t, respectively. These represent the width and height of the first tracking region in frame t, respectively.

[0068] The position of the second tracking region in frame t is:

[0069]

[0070] in, This represents the second tracking region in frame t. Let x and y represent the x and y coordinates of the center point of the second tracking region in the image coordinate system in frame t, respectively. These represent the width and height of the second tracking region in frame t, respectively.

[0071] The motion trajectory of the first tracking area is then represented as:

[0072] ;

[0073] Wherein, T1 represents the motion trajectory of the first tracking region in consecutive video frames of the video stream, used to characterize the motion trajectory of the container. This indicates the position of the first tracking region in frames 1 to t.

[0074] The motion trajectory of the second tracking area is represented as follows:

[0075] ;

[0076] Wherein, T2 represents the motion trajectory of the second tracking region in consecutive video frames of the video stream, used to characterize the motion trajectory of the card tray. This indicates the position of the second tracking region in frames 1 to t.

[0077] Furthermore, in step 5, the three-dimensional determination includes proximal separation determination and distal separation determination; the spacing, angle, and distance change rate are calculated as follows:

[0078] The spacing D t Calculated based on the perpendicular distance between the first tracking region and the corresponding second tracking region in frame t:

[0079] ;

[0080] Among them, D t This represents the distance between the first tracking region and the corresponding second tracking region in frame t. ) represents the position information of the center point of the first tracking region in the image coordinate system in frame t. () represents the position information of the center point of the second tracking region in the image coordinate system in frame t;

[0081] The motion trajectory angle of the second tracking area Calculated based on the displacement of the center point of the second tracking region in adjacent frames:

[0082] ;

[0083] in, This represents the angle of the motion trajectory of the second tracking region in frame t relative to frame t-1. This represents the coordinates of the center point of the second tracking region in frame t. Let represent the coordinates of the center point of the second tracking region in frame t-1, and arctan denote the arctangent function.

[0084] The distance change rate V t Calculated based on the change in spacing between adjacent frames:

[0085] ;

[0086] Among them, V t D represents the rate of change of distance between the first tracking region and the corresponding second tracking region in frame t. t D represents the distance between the first tracking region and the corresponding second tracking region in frame t. t-1 This represents the distance between the first tracking region and the corresponding second tracking region in frame t-1. Indicates the time interval between two adjacent frames;

[0087] The proximal separation determination is as follows:

[0088] When the spacing D between the first tracking region and the corresponding second tracking region t When the distance exceeds the preset near-end separation threshold D0, it is determined that the container and the truck pallet have experienced near-end separation, i.e., the following conditions are met:

[0089] D t >D0;

[0090] At that time, proximal separation is determined to have occurred;

[0091] Among them, D t D0 represents the distance between the first tracking region and the corresponding second tracking region in frame t, and D0 represents the preset near-end separation threshold.

[0092] Based on the determination of proximal separation, a further determination of distal separation is performed. Distal separation is determined to be normal when the following conditions are met simultaneously:

[0093] ;

[0094] ;

[0095] ;

[0096] Among them, D t D1 represents the distance between the first tracking region and the corresponding second tracking region in frame t, and D1 represents the preset far-end separation threshold. This represents the angle of the motion trajectory of the second tracking region in frame t relative to frame t-1. Indicates the preset angle threshold; Vt V0 represents the rate of change of distance between the first tracking region and the corresponding second tracking region in frame t, and V0 represents the preset distance change rate threshold.

[0097] If the three-dimensional judgment conditions are not met simultaneously, it is judged as an abnormal lifting and an alarm signal is triggered.

[0098] The container truck lifting detection system includes:

[0099] The video acquisition unit is used to acquire video streams from the port's operational area in real time.

[0100] The data processing unit is equipped with the target detection model and the KCF target tracking algorithm, and is used to process video frames and obtain the first tracking area where the container opening is located, the second tracking area where the truck pallet is located, and the motion trajectory corresponding to the two tracking areas.

[0101] The tracking analysis module is used to calculate the spacing, angle, and rate of change of distance between two tracking areas;

[0102] The discrimination and early warning module is used to perform three-dimensional judgment based on the distance, angle and distance change rate between two areas, and to trigger an alarm when a lifting safety risk is determined.

[0103] The display and management module is used to show detection results, motion trajectories, and historical records.

[0104] The beneficial effects of this invention are:

[0105] The container crane lifting detection method and system provided by this invention, based on a combination of detection positioning and KCF tracking, firstly, uses a pre-trained YOLO detection model to accurately locate the container openings, avoiding subjective errors from manual observation and significantly improving the accuracy and robustness of the detection. Secondly, it automatically determines the second tracking area using a spatial correlation derivation model between the container openings and the container truck pallet, eliminating the need for independent detection of the container truck pallet, effectively reducing computational overhead and improving the real-time processing capability of the container crane lifting detection system. Thirdly, it introduces the KCF target tracking algorithm during the lifting process to continuously track the two areas, overcoming interference from complex port environmental factors such as changes in lighting, equipment obstruction, and mechanical vibration, ensuring the stability and continuity of the motion trajectory acquisition. Finally, it performs three-dimensional judgment based on spacing, angle, and distance change rate, combined with near-end separation and far-end separation, to accurately identify the normal separation state of the container and promptly trigger alarms in case of abnormal lifting, thereby significantly reducing the false positive and false negative rates and effectively improving the safety monitoring level and automation of port loading and unloading operations. Attached Figure Description

[0106] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0107] Figure 1 This is a flowchart illustrating the truck lifting detection method based on the combination of detection positioning and KCF tracking in Example 1.

[0108] Figure 2 This is a schematic diagram of the dual-region positioning structure in Example 1;

[0109] Among them, 100 is the container, 101 is the container opening, 102 is the first tracking area, 201 is the second tracking area, and 200 is the truck pallet.

[0110] Figure 3 This is a schematic diagram of KCF tracking in Example 1;

[0111] in, Figure 3 'a' represents the KCF tracking of the t-th frame image. Figure 3 b represents the KCF tracking of the (t+1)th frame image. Figure 3 c represents the KCF tracking of the (t+2)th frame image.

[0112] Figure 4 This is a schematic diagram illustrating the lifting status determination of the container truck in Example 1;

[0113] in, Figure 4 a represents normal separation. Figure 4 b indicates an abnormal lifting operation. Detailed Implementation

[0114] The embodiments of the invention will now be described in detail with reference to the accompanying drawings.

[0115] Example 1

[0116] The process of the truck lifting detection method based on the combination of detection and positioning and KCF tracking is as follows: Figure 1 As shown, the method includes the following steps:

[0117] Step 1: Real-time acquisition of video streams of the port operation area, and extraction of key frame images from the video stream according to a preset extraction strategy; the key frame image is an image frame that completely covers at least one container opening and the core detection target of the truck pallet.

[0118] Step 2: Use a pre-trained target detection model to accurately locate the container openings in the keyframe image, and set the area where the located container openings are located as the first tracking area;

[0119] Step 3: Based on the position information of the first tracking area in the image coordinate system of the key frame image, the position of the truck pallet under the container is calculated using a preset spatial correlation derivation model, and the area where the truck pallet is located is set as the second tracking area;

[0120] Step 4: During the container lifting process, the first tracking area and the corresponding second tracking area are used as the initial target templates of the KCF tracker to track the first tracking area and the corresponding second tracking area in the continuous video frames of the video stream, and the motion trajectory of the first tracking area and the corresponding second tracking area is obtained in real time.

[0121] Step 5: Based on the motion trajectory of the first tracking area and the corresponding second tracking area, calculate the distance, angle and distance change rate between the first tracking area and the corresponding second tracking area, and perform three-dimensional determination; wherein, the distance is the perpendicular distance between the first tracking area and the corresponding second tracking area, the angle is the motion trajectory angle of the second tracking area, and the distance change rate is the rate of change of the distance between the first tracking area and the corresponding second tracking area over time.

[0122] Step 6: When the judgment result is abnormal lifting, the truck lifting detection system triggers an alarm signal.

[0123] In this embodiment, video acquisition units such as industrial cameras are first specifically set up in the port operation area. The industrial cameras can be installed near port quay cranes, yard cranes, gantry cranes, trolley frames, spreaders, or on fixed brackets in the port operation area, and their posture and shooting angle remain unchanged during operation. Simultaneously, the field of view of the industrial cameras must cover the area shown in the attached figure. Figure 2 The diagram shows the container 100, container openings 101 (mainly referring to the container openings 101 on the left and right sides of the long side of the container 100), the truck pallet 200, and the separation area between the container 100 and the truck pallet 200. After acquiring the video stream, the truck lifting detection system extracts keyframe images from the video stream according to detection requirements. For example, keyframe images are extracted at fixed frame intervals of 20 frames, and then the extracted keyframe images are used as input images for subsequent target detection models.

[0124] Subsequently, a pre-trained target detection model was used to detect and locate the container opening 101 in the keyframe image. The container opening 101 was chosen as the detection target because it is a relatively stable local target with a clear location and obvious morphological features in the container structure. Compared with directly detecting the entire container or truck, the container opening area is less affected by background interference and has a stable spatial relationship with the lower edge of the container and the position of the truck pallet. Therefore, using the container opening detection result as the first tracking area 102 can provide a more stable initial template for subsequent tracking. After obtaining the first tracking area 102, the present invention does not directly perform independent detection on the truck pallet 200. Instead, it uses a spatial correlation derivation model to calculate the area where the truck pallet 200 is located based on the position of the first tracking area 102 in the image coordinate system, and sets this area as the second tracking area 201. The reason for introducing this step is that the truck pallet 200 is easily affected by container obstruction, vehicle structure obstruction, shadows, reflections and complex backgrounds in the actual port operation area. If the truck pallet is detected separately, not only is the detection difficult, but the amount of calculation will also increase. By using the spatial correlation derivation from the container opening 101 to the truck pallet 200, the performance burden caused by frame-by-frame detection of the truck pallet can be avoided, and the reliability of the truck pallet area initialization can be improved.

[0125] Subsequently, during the container lifting process, as shown in the attached... Figure 3 As shown, the truck lifting detection system uses the first tracking area 102 and the second tracking area 201 as the initial target templates of the two KCF trackers, respectively, to track the two areas in the continuous video frames of the video stream in real time. The KCF target tracking algorithm has the characteristics of fast calculation speed and suitability for real-time video processing. It can quickly obtain the target position change in continuous frames when the target area has been initialized.

[0126] Finally, the truck lifting detection system calculates the distance, angle, and rate of change of distance between the first tracking area 102 and the second tracking area 201 based on their movement trajectories, performing a three-dimensional judgment. Relying solely on distance for judgment may lead to misjudgments during the initial stages of normal container lifting, mechanical vibrations, or image shake. Adding angle judgment allows identification of abnormal tilting or movement of the truck pallet. Furthermore, adding distance rate of change judgment allows determination of whether the separation process between the two areas conforms to normal lifting patterns. Therefore, three-dimensional judgment can more accurately distinguish between, for example, the following... Figure 4 Zhongfu Figure 4 The normal separation state and attachment shown in figure a Figure 4The abnormal lifting status is shown in b. When the judgment result is an abnormal lifting, the truck lifting detection system triggers an alarm signal. The alarm signal can take the form of audible and visual alarm, monitoring platform pop-up window, operation control system linkage prompt, remote terminal push, etc. Through this alarm mechanism, the system can promptly remind on-site personnel or linkage control equipment when the truck is being lifted together or there is a risk of lifting, thereby reducing the probability of safety accidents.

[0127] Furthermore, in step 2, the pre-trained target detection model is used to accurately locate the container openings in the keyframe image, and the area where the located container openings are located is set as the first tracking area. The specific process includes:

[0128] Step 21: Set the target detection model to the YOLO detection model;

[0129] Step 22: Obtain image samples of port operation areas containing container openings, and label the locations of the container openings in the image samples to obtain a container opening training sample set;

[0130] Step 23: Train the YOLO detection model using the container hole training sample set to obtain a pre-trained YOLO detection model;

[0131] Step 24: Input the keyframe image into the pre-trained YOLO detection model, and the pre-trained YOLO detection model outputs the container hole detection box and its detection confidence, as well as the position information of the center point of the container hole detection box in the image coordinate system of the keyframe image;

[0132] Step 25: Sort the detection confidence scores from high to low, and take the container hole detection frames corresponding to the highest sorted detection confidence scores as valid container hole detection results, and determine the first tracking area based on the valid container hole detection results.

[0133] In this embodiment, the target detection model is preferably the YOLO detection model. The YOLO detection model has the characteristics of end-to-end detection, high speed, and suitability for real-time deployment, which can meet the real-time requirements of video processing in port operation areas.

[0134] The YOLO detection model can be YOLOv8, YOLO11, YOLO26, or other YOLO series models with the same real-time object detection capabilities. It can also be a lightweight improvement on YOLO or a detection model optimized for container hole targets. It should be noted that the detection model capable of locating container holes is not limited to the YOLO series. Other mainstream object detection models are also applicable, such as SSD, Faster R-CNN, CenterNet, EfficientDet, and lightweight models suitable for edge computing devices, such as MobileNet-SSD or Nanodet. In actual deployment, any of the above models or their improved versions for container hole detection tasks can be flexibly selected according to computing resources, real-time requirements, and detection accuracy requirements.

[0135] To enable the YOLO detection model to adapt to the complex environment of the port operation area, image samples of the port operation area containing container openings 101 were first collected. The samples should cover different lighting conditions, different weather conditions, different container colors, different camera angles, different operating distances, different occlusion situations, and different background complexities. Then, the positions of the container openings in the samples were manually labeled to form a training sample set of container openings.

[0136] The YOLO detection model is trained using a training sample set of container holes, enabling the model to learn the shape, edge, texture, and spatial location features of container hole 101. After training, keyframe images are input into the pre-trained YOLO detection model, and the model outputs one or more container hole detection boxes, corresponding detection confidence scores, and the coordinates of the center point of the detection box.

[0137] Since there may be interfering targets such as keyholes, vehicle structure holes, and shadow holes in the port image, this embodiment further sorts them according to the detection confidence level, and takes the detection boxes with the highest ranking as the effective container box hole detection results, so as to provide a stable and accurate initial position for the subsequent generation of the first tracking area 102.

[0138] Further, in step 25, determining the first tracking area based on the effective container opening detection results specifically includes:

[0139] The effective container opening detection results are expressed as follows:

[0140] B h =(x h ,y h ,w h ,h h );

[0141] Among them, B hx represents the container hole detection bounding box output by the pre-trained YOLO detection model. h y h These represent the x and y coordinates of the center point of the container opening detection frame in the image coordinate system of the keyframe image, respectively. h h h These represent the width and height of the container opening detection frame, respectively.

[0142] The first tracking area is generated based on the container hole detection frame:

[0143]

[0144] in:

[0145] x1=x h ;

[0146] y1=y h ;

[0147] ;

[0148] ;

[0149] Where R1 represents the first tracking region, x1 and y1 represent the x and y coordinates of the center point of the first tracking region in the image coordinate system of the keyframe image, respectively, and w1 and h1 represent the width and height of the first tracking region, respectively; x h y h These represent the x and y coordinates of the center point of the container opening detection frame in the image coordinate system of the keyframe image, respectively; w h h h These represent the width and height of the container opening inspection frame, respectively. Indicates the horizontal expansion coefficient. This represents the vertical expansion coefficient, and , .

[0150] In this embodiment, the container opening detection bounding box output by the pre-trained YOLO detection model typically only covers the container opening 101 itself. Since the area of ​​the container opening 101 is small, and in real-world scenarios, edge blurring, light reflection, partial occlusion, or image compression noise may occur, directly using the container opening detection bounding box as the KCF tracking template may result in insufficient tracking features, leading to KCF tracking drift or loss. Therefore, this invention appropriately extends the effective container opening detection bounding box to generate a first tracking region 102.

[0151] Specifically, the center point of the first tracking area 102 is consistent with the center point of the container hole detection frame. The expanded first tracking area 102 includes not only the container hole 101, but also the local texture, edge and structural information of the container around the container hole. Through this expansion process, the first tracking area 102 has richer image features, which is conducive to the KCF tracker to build a more stable target template.

[0152] Further, in step 3, based on the position information of the first tracking area in the image coordinate system of the keyframe image, the position of the truck pallet below the container is calculated using a preset spatial correlation derivation model, and the area where the truck pallet is located is set as the second tracking area. The specific process includes:

[0153] Step 31: Obtain the coordinates of the center point, width, and height of the first tracking region in the image coordinate system;

[0154] Step 32: Call the preset spatial correlation derivation model, which takes the center point coordinates, width, height and preset spatial correlation parameters of the first tracking area as input;

[0155] Step 33: Output the center point coordinates, width, and height of the second tracking region from the spatial correlation derivation model;

[0156] Step 34: Determine the area where the truck tray is located in the keyframe image based on the center point coordinates, area width, and area height of the second tracking area.

[0157] In this embodiment, after obtaining the first tracking region 102, the system further obtains the center point coordinates, width, and height of the region in the image coordinate system. Since in the port operation scenario, the container 100 is placed above the truck pallet 200, and from the same camera perspective, the container opening 101 and the truck pallet 200 have a relatively stable positional relationship, the second tracking region 201 can be calculated using a spatial correlation derivation model.

[0158] The spatial correlation derivation model receives the position parameters of the first tracking region 102 and preset spatial correlation parameters, and outputs the center point coordinates, width, and height of the second tracking region 201. Subsequently, the system determines the region where the truck pallet 200 is located in the keyframe image according to the output results. Existing technologies, if the truck pallet 200 is directly detected, are easily affected by pallet color, truck structure, shadows, occlusion, and background complexity, and the computational load is large. This embodiment derives the pallet region from the box hole detection results, reducing the need for independent target detection of the truck pallet 200, reducing system computational overhead, and establishing a spatial correspondence between the first tracking region 102 and the second tracking region 201, providing a unified coordinate basis for subsequent calculations of the distance, angle, and distance change rate between the two.

[0159] Furthermore, in step 3, when the target detection model detects different numbers of container openings in the keyframe image, it determines the position for deriving the center point of the second tracking region in different ways. The specific process includes:

[0160] When the target detection model detects only one container opening in the keyframe image, the recognition process obtains the first tracking region of the container opening. Then, the spatial association derivation model derives the position of the second tracking region based on the first tracking region corresponding to the single container opening, specifically including:

[0161] Obtain the center point of the first tracking area corresponding to the container opening:

[0162] C1=(x1,y1);

[0163] Wherein, C1 represents the center point of the first tracking area, and x1 and y1 represent the horizontal and vertical coordinates of the center point of the first tracking area corresponding to the container opening in the image coordinate system of the keyframe image, respectively.

[0164] The center point of the second tracking area is:

[0165] ;

[0166] Where C2 represents the center point of the second tracking area, x1 and y1 represent the horizontal and vertical coordinates of the center point of the first tracking area corresponding to the container opening in the image coordinate system of the keyframe image, respectively, kx represents the preset horizontal offset parameter, and ky represents the preset vertical offset parameter;

[0167] When the target detection model simultaneously detects multiple container openings on the same side in the keyframe image, it selects the two container openings with the highest detection confidence scores and performs recognition processing on the left and right container openings in the keyframe image to obtain the first tracking region corresponding to the left container opening and the first tracking region corresponding to the right container opening. Then, the spatial association derivation model deduces the position of the corresponding second tracking region based on the first tracking regions corresponding to the two container openings, specifically including:

[0168] Obtain the center point of the first tracking area corresponding to the opening of the left container:

[0169] C 1L =(x 1L ,y 1L );

[0170] Among them, C 1L x represents the center point of the first tracking area corresponding to the opening of the left container. 1L y 1L These represent the x and y coordinates of the center point of the first tracking area corresponding to the left container opening in the image coordinate system of the keyframe image, respectively.

[0171] And the center point of the first tracking area corresponding to the right container opening:

[0172] C 1R =(x 1R ,y 1R );

[0173] Among them, C 1R x represents the center point of the first tracking area corresponding to the right container opening. 1R y 1R These represent the x and y coordinates of the center point of the first tracking area corresponding to the container opening on the right in the image coordinate system of the keyframe image, respectively.

[0174] The reference position of the lower edge of the container is determined based on two center points, and the center point of the second tracking area is determined according to the preset horizontal offset parameter kx and the preset vertical offset parameter ky.

[0175] ;

[0176] Where C2 represents the center point of the second tracking region, x 1L y 1L These represent the x and y coordinates of the center point of the first tracking area corresponding to the opening of the left container in the image coordinate system of the keyframe image, respectively. 1R y 1RThese represent the x and y coordinates of the center point of the first tracking area corresponding to the right container opening in the image coordinate system of the keyframe image, respectively. kx represents the preset horizontal offset parameter, and ky represents the preset vertical offset parameter.

[0177] In actual port operation videos, due to differences in shooting angle, container obstruction, truck position changes, or lighting conditions, keyframe images may detect only one container opening 101, or multiple container openings 101 may be detected simultaneously. When only one container opening 101 is detected, the system uses the first tracking area 102 corresponding to that container opening as a reference and calculates the center point of the second tracking area 201 using preset horizontal and vertical offset parameters.

[0178] When multiple container openings are detected simultaneously, the two container openings with the highest detection confidence levels are selected, namely the left and right container openings 101. The truck lifting detection system acquires the center points of the left and right first tracking areas respectively, and takes the average position of the two center points as the reference position of the lower edge of the container. The center point of the second tracking area 201 is determined by preset horizontal and vertical offset parameters. This method uses the two openings to jointly constrain the pallet position, which can reduce the positional deviation caused by false detection of a single opening or partial obstruction.

[0179] Furthermore, the spatial correlation derivation model is constructed in the following manner:

[0180] Multiple calibration sample images are collected from the same camera viewpoint within the port operation area. The calibration sample images include container openings and corresponding truck pallets, and the container opening area and the corresponding truck pallet area are marked in the calibration sample images respectively.

[0181] Based on the center point coordinates, width, and height of the container opening area, and the center point coordinates, width, and height of the truck pallet area, calculate the spatial correlation parameters of the truck pallet area relative to the container opening area. The spatial correlation parameters include at least horizontal offset parameters, vertical offset parameters, width ratio parameters, and height ratio parameters.

[0182] Statistical processing is performed on the spatial correlation parameters corresponding to multiple frames of the calibration sample images to obtain preset spatial correlation parameters; the preset spatial correlation parameters are configured into the initial spatial correlation derivation model to obtain the spatial correlation derivation model, so that the spatial correlation derivation model can output the center point coordinates, region width, and region height of the second tracking region corresponding to the first tracking region based on the input center point coordinates, region width, and region height of the first tracking region.

[0183] In the actual detection process, the spatial correlation derivation model derives the center point coordinates, width, and height of the second tracking area based on the center point coordinates, width, and height of the first tracking area, as well as the preset spatial correlation parameters.

[0184] Wherein, the first tracking region is represented as:

[0185] R1 = (x1, y1, w1, h1);

[0186] Wherein, R1 represents the first tracking region, x1 and y1 represent the horizontal and vertical coordinates of the center point of the first tracking region in the image coordinate system of the keyframe image, respectively, and w1 and h1 represent the width and height of the first tracking region, respectively.

[0187] The second tracking region is represented as:

[0188] R2 = (x2, y2, w2, h2);

[0189] Where R2 represents the second tracking region, x2 and y2 represent the x and y coordinates of the center point of the second tracking region in the image coordinate system of the keyframe image, respectively, and w2 and h2 represent the width and height of the second tracking region, respectively.

[0190] but:

[0191]

[0192] ;

[0193] ;

[0194] ;

[0195] Where kx represents the preset horizontal offset parameter, ky represents the preset vertical offset parameter, kw represents the preset width ratio parameter, and kh represents the preset height ratio parameter.

[0196] In this embodiment, the spatial correlation derivation model is obtained through statistical analysis of multiple calibration sample images from the same camera's perspective. Specifically, multiple sample images taken by a fixed camera within the port operation area are selected, and these images simultaneously contain container openings 101 and truck pallets 200. The opening area and pallet area are labeled manually or semi-automatically.

[0197] For each frame of the calibration sample image, the horizontal offset parameter, vertical offset parameter, width ratio parameter, and height ratio parameter of the pallet area relative to the box hole area are calculated. Subsequently, the parameters corresponding to multiple frames of samples are statistically processed to obtain preset spatial association parameters applicable to the camera's viewpoint. In the actual detection process, after the first tracking area 102 is detected and located, the spatial association derivation model outputs the second tracking area 201 according to the preset parameters.

[0198] Further, in step 4, KCF tracking is performed on the first tracking region and the corresponding second tracking region. The specific process includes:

[0199] Step 41: Set the first tracking region as the initial target template of the first KCF tracker, and the second tracking region as the initial target template of the second KCF tracker;

[0200] Step 42: Calculate the correlation response maps of the first KCF tracker and the second KCF tracker in consecutive video frames of the video stream, respectively;

[0201] Step 43: Determine the positions corresponding to the pixels with the largest response values ​​in the relevant response map as the positions of the first tracking region and the corresponding second tracking region in the current frame, respectively;

[0202] Step 44: Update the target template of the corresponding KCF tracker according to the position of the first tracking region and the corresponding position of the second tracking region in the current frame;

[0203] Step 45: When the maximum response value of the correlation response map of the two KCF trackers is greater than or equal to the preset response threshold, the position corresponding to the pixel with the largest response value in the correlation response map is determined as the first tracking region position and the corresponding second tracking region position in the current frame, respectively; when the maximum response value of the correlation response map of any KCF tracker is lower than the preset response threshold, the target detection model is called again to locate the container opening, and the first tracking region and the corresponding second tracking region are re-initialized through the spatial association derivation model.

[0204] In this embodiment, as shown in the appendix Figure 3 As shown, after determining the first tracking region 102 and the second tracking region 201 in the keyframe image, the truck lifting detection system establishes two KCF trackers respectively. The first KCF tracker is used to track the first tracking region 102 where the container opening 101 is located, thereby characterizing the movement state of the container 100; the second KCF tracker is used to track the second tracking region 201 where the truck pallet 200 is located, thereby characterizing the movement state of the truck pallet 200. (See attached image) Figure 3As shown in Figure a, in frame t, the first tracking region 102 and the second tracking region 201 are initialized or updated respectively; as attached. Figure 3 As shown in b, in frame t+1, the KCF tracker continues to output the positions of the first tracking region 102 and the second tracking region 201; as attached. Figure 3 As shown in Figure c, in frame t+2, the truck lifting detection system continues to acquire the changed positions of the two areas. The movement trajectories of the container 100 and the truck pallet 200 can be formed through the position changes in consecutive frames.

[0205] The KCF target tracking algorithm learns an initial target template and calculates a correlation response map in subsequent video frames. A higher response value in the correlation response map indicates a higher probability that the current location is the target location. Therefore, the truck lifting detection system determines the location of the pixel with the highest response value as the target region location in the current frame and updates the target template accordingly.

[0206] To avoid long-term tracking drift, this embodiment introduces a preset response threshold. When the maximum response values ​​of both KCF trackers are greater than or equal to the preset response threshold, the tracking is considered reliable. When the maximum response value of either KCF tracker is lower than the preset response threshold, it indicates that the target may be occluded, the tracking may be drifting, or the image may have changed abruptly. In this case, the system re-calls the YOLO detection model to locate the container opening 101 and re-initializes the first tracking area 102 and the second tracking area 201 through the spatial correlation derivation model.

[0207] Further, in step 4, the motion trajectory includes the position sequence of the first tracking region in consecutive video frames of the video stream and the position sequence of the second tracking region in consecutive video frames of the video stream;

[0208] Let the position of the first tracking region in frame t be:

[0209]

[0210] Where t represents the video frame number. This represents the first tracking region in frame t. Let x and y represent the x and y coordinates of the center point of the first tracking region in the image coordinate system in frame t, respectively. These represent the width and height of the first tracking region in frame t, respectively.

[0211] The position of the second tracking region in frame t is:

[0212]

[0213] in, This represents the second tracking region in frame t. Let x and y represent the x and y coordinates of the center point of the second tracking region in the image coordinate system in frame t, respectively. These represent the width and height of the second tracking region in frame t, respectively.

[0214] The motion trajectory of the first tracking area is then represented as:

[0215] ;

[0216] Wherein, T1 represents the motion trajectory of the first tracking region in consecutive video frames of the video stream, used to characterize the motion trajectory of the container. This indicates the position of the first tracking region in frames 1 to t.

[0217] The motion trajectory of the second tracking area is represented as follows:

[0218] ;

[0219] Wherein, T2 represents the motion trajectory of the second tracking region in consecutive video frames of the video stream, used to characterize the motion trajectory of the card tray. This indicates the position of the second tracking region in frames 1 to t.

[0220] In this embodiment, the KCF tracker outputs the positions of the first tracking region 102 and the second tracking region 201 in each frame. The truck lifting detection system uses the position parameters obtained in consecutive frames to assemble the motion trajectories of the first tracking region and the second tracking region, respectively.

[0221] The motion trajectory of the first tracking area is used to characterize the motion trajectory of the container 100. The motion trajectory of the second tracking area is used to characterize the motion trajectory of the truck pallet 200. Under normal circumstances, the truck pallet 200 should remain relatively stationary or experience only slight vibrations; if the motion trajectory of the second tracking area shows that the second tracking area 201 rises synchronously with the motion trajectory of the first tracking area or undergoes significant tilting movement, there may be a risk of abnormal lifting. By constructing continuous frame positions as a trajectory sequence, this invention no longer relies on single-frame images for judgment, but analyzes dynamic changes in the time dimension, enabling subsequent three-dimensional judgments to be based on the actual motion process, rather than on a static distance at a certain instant.

[0222] Furthermore, in step 5, the three-dimensional determination includes proximal separation determination and distal separation determination; the spacing, angle, and distance change rate are calculated as follows:

[0223] The spacing D t Calculated based on the perpendicular distance between the first tracking region and the corresponding second tracking region in frame t:

[0224] ;

[0225] Among them, D t This represents the distance between the first tracking region and the corresponding second tracking region in frame t. ) represents the position information of the center point of the first tracking region in the image coordinate system in frame t. () represents the position information of the center point of the second tracking region in the image coordinate system in frame t;

[0226] The motion trajectory angle of the second tracking area Calculated based on the displacement of the center point of the second tracking region in adjacent frames:

[0227] ;

[0228] in, This represents the angle of the motion trajectory of the second tracking region in frame t relative to frame t-1. This represents the coordinates of the center point of the second tracking region in frame t. Let represent the coordinates of the center point of the second tracking region in frame t-1, and arctan denote the arctangent function.

[0229] The distance change rate V t Calculated based on the change in spacing between adjacent frames:

[0230] ;

[0231] Among them, V t D represents the rate of change of distance between the first tracking region and the corresponding second tracking region in frame t. t D represents the distance between the first tracking region and the corresponding second tracking region in frame t. t-1 This represents the distance between the first tracking region and the corresponding second tracking region in frame t-1. Indicates the time interval between two adjacent frames;

[0232] The proximal separation determination is as follows:

[0233] When the spacing D between the first tracking region and the corresponding second tracking region t When the distance exceeds the preset near-end separation threshold D0, it is determined that the container and the truck pallet have experienced near-end separation, i.e., the following conditions are met:

[0234] D t >D0;

[0235] At that time, proximal separation is determined to have occurred;

[0236] Among them, D tD0 represents the distance between the first tracking region and the corresponding second tracking region in frame t, and D0 represents the preset near-end separation threshold.

[0237] Based on the determination of proximal separation, a further determination of distal separation is performed. Distal separation is determined to be normal when the following conditions are met simultaneously:

[0238] ;

[0239] ;

[0240] ;

[0241] Among them, D t D1 represents the distance between the first tracking region and the corresponding second tracking region in frame t, and D1 represents the preset far-end separation threshold. This represents the angle of the motion trajectory of the second tracking region in frame t relative to frame t-1. Indicates the preset angle threshold; V t V0 represents the rate of change of distance between the first tracking region and the corresponding second tracking region in frame t, and V0 represents the preset distance change rate threshold.

[0242] If the three-dimensional judgment conditions are not met simultaneously, it is judged as an abnormal lifting and an alarm signal is triggered.

[0243] In this embodiment, the three-dimensional determination includes spacing, angle, and distance change rate. Spacing can only determine whether the container 100 and the truck pallet 200 have separated at a certain moment, but it cannot determine whether the pallet has been lifted, whether it has tilted, or whether there is an abnormal situation where one end is not separated. Therefore, this embodiment introduces three-dimensional determination to improve the accuracy of abnormal lifting identification.

[0244] The truck lifting detection system calculates the distance between the first tracking area 102 and the second tracking area 201 to reflect the degree of spatial separation between the container 100 and the truck pallet 200. During normal lifting, the distance should gradually increase as the container 100 rises. If the distance increases abnormally, there may be a problem where the truck pallet 200 is being lifted. The truck lifting detection system calculates the motion trajectory angle of the second tracking area 201 to reflect the direction of movement of the truck pallet 200. Under normal circumstances, the truck pallet 200 should remain basically horizontal and stationary. If the pallet is lifted at one end or tilted, the direction of movement of the center point of the second tracking area 201 will show an abnormal angle change, as shown in the attached figure. Figure 4As shown in b; the truck lifting detection system calculates the distance change rate to reflect the rate of increase in distance between container 100 and truck pallet 200; during normal separation, the distance change rate should be greater than the preset threshold; if container 100 and truck pallet 200 rise synchronously, the distance change rate between them may be small, or even close to zero.

[0245] In this embodiment, the three-dimensional determination adopts a logic of first separating the near end and then separating the far end. First, it checks whether the distance is greater than a preset near-end separation threshold. If not, it indicates that the container 100 and the truck pallet 200 have not yet achieved effective separation or there is a risk of adhesion. If the near-end separation condition is met, it further checks whether the far-end normal separation condition is met. When both conditions are met simultaneously, it is determined to be a far-end normal separation, such as... Figure 4 As shown in a; if any condition is not met, it is judged as an abnormal lifting, such as Figure 4 As shown in b, it triggers an alarm signal.

[0246] Example 2

[0247] The container truck lifting detection system includes:

[0248] The video acquisition unit is used to acquire video streams from the port's operational area in real time.

[0249] The data processing unit is equipped with the target detection model and the KCF target tracking algorithm, and is used to process video frames and obtain the first tracking area where the container opening is located, the second tracking area where the truck pallet is located, and the motion trajectory corresponding to the two tracking areas.

[0250] The tracking analysis module is used to calculate the spacing, angle, and rate of change of distance between two tracking areas;

[0251] The discrimination and early warning module is used to perform three-dimensional judgment based on the distance, angle and distance change rate between two areas, and to trigger an alarm when a lifting safety risk is determined.

[0252] The display and management module is used to show detection results, motion trajectories, and historical records.

[0253] In this embodiment, a truck lifting detection system is used to implement the above method. The truck lifting detection system can be deployed on a local port server, edge computing device, or intelligent monitoring platform.

[0254] The video acquisition unit includes, but is not limited to, cameras, video capture cards, network transmission modules, or video stream access interfaces, used to acquire video streams of the port operation area in real time. The cameras are preferably installed in positions that cover the bottom of the container 100 (container opening 101) and the truck pallet 200, ensuring that keyframe images completely cover the core detection target.

[0255] The data processing unit is equipped with a target detection model, a spatial correlation derivation model, and a KCF target tracking algorithm. The unit first uses the target detection model to locate the container opening 101 and generates a first tracking area 102. Then, it uses the spatial correlation derivation model to calculate a second tracking area 201 where the container pallet 200 is located. Subsequently, it uses the KCF target tracking algorithm to continuously track the first tracking area 102 and the second tracking area 201, outputting the motion trajectories corresponding to the two tracking areas.

[0256] The tracking and analysis module calculates parameters such as spacing, angle, and distance change rate based on the motion trajectory. The discrimination and early warning module determines whether the current lifting status is normal separation or abnormal lifting based on three-dimensional judgment rules. When abnormal lifting is determined, the discrimination and early warning module can trigger alarms including but not limited to audible and visual alarms, control room pop-up alarms, SMS alarms, platform push alarms, or send stop or deceleration prompt signals to the lifting equipment control system.

[0257] The display and management module displays real-time video feeds, the first tracking area 102, the second tracking area 201, the motion trajectory, the judgment result, alarm time, and historical records. Through this module, administrators can trace the occurrence of abnormal events, providing a basis for safety management and accident analysis.

Claims

1. A method for detecting the lifting of container trucks based on a combination of detection and positioning and KCF tracking, characterized in that, The method includes the following steps: Step 1: Real-time acquisition of video streams of the port operation area, and extraction of key frame images from the video stream according to a preset extraction strategy; the key frame image is an image frame that completely covers at least one container opening and the core detection target of the truck pallet. Step 2: Use a pre-trained target detection model to accurately locate the container openings in the keyframe image, and set the area where the located container openings are located as the first tracking area; Step 3: Based on the position information of the first tracking area in the image coordinate system of the key frame image, the position of the truck pallet under the container is calculated using a preset spatial correlation derivation model, and the area where the truck pallet is located is set as the second tracking area; Step 4: During the container lifting process, the first tracking area and the corresponding second tracking area are used as the initial target templates of the KCF tracker to track the first tracking area and the corresponding second tracking area in the continuous video frames of the video stream, and the motion trajectory of the first tracking area and the corresponding second tracking area is obtained in real time. Step 5: Based on the motion trajectory of the first tracking area and the corresponding second tracking area, calculate the distance, angle and distance change rate between the first tracking area and the corresponding second tracking area, and perform three-dimensional determination; wherein, the distance is the perpendicular distance between the first tracking area and the corresponding second tracking area, the angle is the motion trajectory angle of the second tracking area, and the distance change rate is the rate of change of the distance between the first tracking area and the corresponding second tracking area over time. Step 6: When the judgment result is abnormal lifting, the truck lifting detection system triggers an alarm signal.

2. The truck lifting detection method based on the combination of detection positioning and KCF tracking according to claim 1, characterized in that, The specific process in step 2 includes: Step 21: Set the target detection model to the YOLO detection model; Step 22: Obtain image samples of port operation areas containing container openings, and label the locations of the container openings in the image samples to obtain a container opening training sample set; Step 23: Train the YOLO detection model using the container hole training sample set to obtain a pre-trained YOLO detection model; Step 24: Input the keyframe image into the pre-trained YOLO detection model, and the pre-trained YOLO detection model outputs the container hole detection box and its detection confidence, as well as the position information of the center point of the container hole detection box in the image coordinate system of the keyframe image; Step 25: Sort the detection confidence scores from high to low, and take the container hole detection frames corresponding to the highest sorted detection confidence scores as valid container hole detection results, and determine the first tracking area based on the valid container hole detection results.

3. The truck lifting detection method based on the combination of detection positioning and KCF tracking according to claim 2, characterized in that, In step 25, determining the first tracking area based on the effective container opening detection results specifically includes: The effective container opening detection results are expressed as follows: B h =(x h ,y h ,w h ,h h ); Among them, B h x represents the container hole detection bounding box output by the pre-trained YOLO detection model. h y h These represent the x and y coordinates of the center point of the container opening detection frame in the image coordinate system of the keyframe image, respectively. h h h These represent the width and height of the container opening detection frame, respectively. The first tracking area is generated based on the container hole detection frame: in: x1=x h ; y1=y h ; ; ; Where R1 represents the first tracking region, x1 and y1 represent the x and y coordinates of the center point of the first tracking region in the image coordinate system of the keyframe image, respectively, and w1 and h1 represent the width and height of the first tracking region, respectively; x h y h These represent the x and y coordinates of the center point of the container opening detection frame in the image coordinate system of the keyframe image, respectively; w h h h These represent the width and height of the container opening inspection frame, respectively. Indicates the horizontal expansion coefficient. This represents the vertical expansion coefficient, and , .

4. The truck lifting detection method based on the combination of detection positioning and KCF tracking according to claim 1, characterized in that, The specific process in step 3 includes: Step 31: Obtain the coordinates of the center point, width, and height of the first tracking region in the image coordinate system; Step 32: Call the preset spatial correlation derivation model, which takes the center point coordinates, width, height and preset spatial correlation parameters of the first tracking area as input; Step 33: Output the center point coordinates, width, and height of the second tracking region from the spatial correlation derivation model; Step 34: Determine the area where the truck tray is located in the keyframe image based on the center point coordinates, area width, and area height of the second tracking area.

5. The truck lifting detection method based on the combination of detection positioning and KCF tracking according to claim 4, characterized in that, In step 3, when the target detection model detects different numbers of container openings in the keyframe image, it determines the position for deriving the center point of the second tracking area in different ways. The specific process includes: When the target detection model detects only one container opening in the keyframe image, the recognition process obtains the first tracking region of the container opening. Then, the spatial association derivation model derives the position of the second tracking region based on the first tracking region corresponding to the single container opening, specifically including: Obtain the center point of the first tracking area corresponding to the container opening: C1=(x1,y1); Where C1 represents the center point of the first tracking area, and x1 and y1 represent the horizontal and vertical coordinates of the center point of the first tracking area corresponding to the container opening in the image coordinate system of the keyframe image, respectively. The center point of the second tracking area is: ; Where C2 represents the center point of the second tracking area, x1 and y1 represent the horizontal and vertical coordinates of the center point of the first tracking area corresponding to the container opening in the image coordinate system of the keyframe image, respectively, kx represents the preset horizontal offset parameter, and ky represents the preset vertical offset parameter; When the target detection model simultaneously detects multiple container openings on the same side in the keyframe image, it selects the two container openings with the highest detection confidence scores and performs recognition processing on the left and right container openings in the keyframe image to obtain the first tracking region corresponding to the left container opening and the first tracking region corresponding to the right container opening. Then, the spatial association derivation model deduces the position of the corresponding second tracking region based on the first tracking regions corresponding to the two container openings, specifically including: Obtain the center point of the first tracking area corresponding to the opening of the left container: C 1L =(x 1L ,y 1L ); Among them, C 1L x represents the center point of the first tracking area corresponding to the opening of the left container. 1L y 1L These represent the x and y coordinates of the center point of the first tracking area corresponding to the left container opening in the image coordinate system of the keyframe image, respectively. And the center point of the first tracking area corresponding to the right container opening: C 1R =(x 1R ,y 1R ); Among them, C 1R x represents the center point of the first tracking area corresponding to the right container opening. 1R y 1R These represent the x and y coordinates of the center point of the first tracking area corresponding to the container opening on the right in the image coordinate system of the keyframe image, respectively. The reference position of the lower edge of the container is determined based on two center points, and the center point of the second tracking area is determined according to the preset horizontal offset parameter kx and the preset vertical offset parameter ky. ; Where C2 represents the center point of the second tracking region, x 1L y 1L These represent the x and y coordinates of the center point of the first tracking area corresponding to the opening of the left container in the image coordinate system of the keyframe image, respectively. 1R y 1R These represent the x and y coordinates of the center point of the first tracking area corresponding to the right container opening in the image coordinate system of the keyframe image, respectively. kx represents the preset horizontal offset parameter, and ky represents the preset vertical offset parameter.

6. The truck lifting detection method based on the combination of detection positioning and KCF tracking according to claim 4 or 5, characterized in that, The spatial correlation derivation model is constructed in the following manner: Multiple calibration sample images are collected from the same camera viewpoint within the port operation area. The calibration sample images include container openings and corresponding truck pallets, and the container opening area and the corresponding truck pallet area are marked in the calibration sample images respectively. Based on the center point coordinates, width, and height of the container opening area, and the center point coordinates, width, and height of the truck pallet area, calculate the spatial correlation parameters of the truck pallet area relative to the container opening area. The spatial correlation parameters include at least horizontal offset parameters, vertical offset parameters, width ratio parameters, and height ratio parameters. Statistical processing is performed on the spatial correlation parameters corresponding to multiple frames of the calibration sample images to obtain preset spatial correlation parameters; the preset spatial correlation parameters are configured into the initial spatial correlation derivation model to obtain the spatial correlation derivation model, so that the spatial correlation derivation model can output the center point coordinates, region width, and region height of the second tracking region corresponding to the first tracking region based on the input center point coordinates, region width, and region height of the first tracking region.

7. The truck lifting detection method based on the combination of detection positioning and KCF tracking according to claim 1, characterized in that, In step 4, KCF tracking is performed on the first tracking region and the corresponding second tracking region. The specific process includes: Step 41: Set the first tracking region as the initial target template of the first KCF tracker, and the second tracking region as the initial target template of the second KCF tracker; Step 42: Calculate the correlation response maps of the first KCF tracker and the second KCF tracker in consecutive video frames of the video stream, respectively; Step 43: Determine the positions corresponding to the pixels with the largest response values ​​in the relevant response map as the positions of the first tracking region and the corresponding second tracking region in the current frame, respectively; Step 44: Update the target template of the corresponding KCF tracker according to the position of the first tracking region and the corresponding position of the second tracking region in the current frame; Step 45: When the maximum response value of the correlation response map of the two KCF trackers is greater than or equal to the preset response threshold, the position corresponding to the pixel with the largest response value in the correlation response map is determined as the first tracking region position and the corresponding second tracking region position in the current frame, respectively; when the maximum response value of the correlation response map of any KCF tracker is lower than the preset response threshold, the target detection model is called again to locate the container opening, and the first tracking region and the corresponding second tracking region are re-initialized through the spatial association derivation model.

8. The truck lifting detection method based on the combination of detection positioning and KCF tracking according to claim 7, characterized in that, In step 4, the motion trajectory includes the position sequence of the first tracking region in consecutive video frames of the video stream and the position sequence of the second tracking region in consecutive video frames of the video stream. Let the position of the first tracking region in frame t be: Where t represents the video frame number. This represents the first tracking region in frame t. Let x and y represent the x and y coordinates of the center point of the first tracking region in the image coordinate system in frame t, respectively. These represent the width and height of the first tracking region in frame t, respectively. The position of the second tracking region in frame t is: ; in, This represents the second tracking region in frame t. Let x and y represent the x and y coordinates of the center point of the second tracking region in the image coordinate system in frame t, respectively. These represent the width and height of the second tracking region in frame t, respectively. The motion trajectory of the first tracking area is then represented as: ; Wherein, T1 represents the motion trajectory of the first tracking region in consecutive video frames of the video stream, used to characterize the motion trajectory of the container. This indicates the position of the first tracking region in frames 1 to t. The motion trajectory of the second tracking area is represented as follows: ; Wherein, T2 represents the motion trajectory of the second tracking region in consecutive video frames of the video stream, used to characterize the motion trajectory of the card tray. This indicates the position of the second tracking region in frames 1 to t.

9. The truck lifting detection method based on the combination of detection positioning and KCF tracking according to claim 1, characterized in that, In step 5, the three-dimensional determination includes proximal separation determination and distal separation determination; the spacing, angle, and distance change rate are calculated as follows: The spacing D t Calculated based on the perpendicular distance between the first tracking region and the corresponding second tracking region in frame t: ; Among them, D t This represents the distance between the first tracking region and the corresponding second tracking region in frame t. ) represents the position information of the center point of the first tracking region in the image coordinate system in frame t. () represents the position information of the center point of the second tracking region in the image coordinate system in frame t; The motion trajectory angle of the second tracking area Calculated based on the displacement of the center point of the second tracking region in adjacent frames: ; in, This represents the angle of the motion trajectory of the second tracking region in frame t relative to frame t-1. This represents the coordinates of the center point of the second tracking region in frame t. Let represent the coordinates of the center point of the second tracking region in frame t-1, and arctan denote the arctangent function. The distance change rate V t Calculated based on the change in spacing between adjacent frames: ; Among them, V t D represents the rate of change of distance between the first tracking region and the corresponding second tracking region in frame t. t D represents the distance between the first tracking region and the corresponding second tracking region in frame t. t-1 This represents the distance between the first tracking region and the corresponding second tracking region in frame t-1. Indicates the time interval between two adjacent frames; The proximal separation determination is as follows: When the spacing D between the first tracking region and the corresponding second tracking region t When the distance exceeds the preset near-end separation threshold D0, it is determined that the container and the truck pallet have experienced near-end separation, i.e., the following conditions are met: D t >D0; At that time, proximal separation is determined to have occurred; Among them, D t D0 represents the distance between the first tracking region and the corresponding second tracking region in frame t, and D0 represents the preset near-end separation threshold. Based on the determination of proximal separation, a further determination of distal separation is performed. Distal separation is determined to be normal when the following conditions are met simultaneously: ; ; ; Among them, D t D1 represents the distance between the first tracking region and the corresponding second tracking region in frame t, and D1 represents the preset far-end separation threshold. This represents the angle of the motion trajectory of the second tracking region in frame t relative to frame t-1. Indicates the preset angle threshold; V t V0 represents the rate of change of distance between the first tracking region and the corresponding second tracking region in frame t, and V0 represents the preset distance change rate threshold. If the three-dimensional judgment conditions are not met simultaneously, it is judged as an abnormal lifting and an alarm signal is triggered.

10. A truck lifting detection system for implementing the truck lifting detection method based on the combination of detection positioning and KCF tracking as described in any one of claims 1-9, characterized in that, include: The video acquisition unit is used to acquire video streams from the port's operational area in real time. The data processing unit is equipped with the target detection model and the KCF target tracking algorithm, and is used to process video frames and obtain the first tracking area where the container opening is located, the second tracking area where the truck pallet is located, and the motion trajectory corresponding to the two tracking areas. The tracking analysis module is used to calculate the spacing, angle, and rate of change of distance between two tracking areas; The discrimination and early warning module is used to perform three-dimensional judgment based on the distance, angle and distance change rate between two areas, and to trigger an alarm when a lifting safety risk is determined. The display and management module is used to show detection results, motion trajectories, and historical records.

Images