Port container cargo loading and unloading state identification method based on computer vision

Abstract

The invention discloses a port container cargo loading and unloading state identification method based on computer vision, and the method comprises the steps: collecting visible light and infrared video images in a port, and carrying out the preprocessing of the images through combining with a defogging algorithm; carrying out the recognition and classification of port objects in the video based on an improved OfficientDet target detection model, and dynamically adjusting the feature fusion weight; in combination with a motion detection technology, if a target is detected to move, the target is divided into a truck type; otherwise, classifying into a'container 'class; further optimizing and classifying the detection target based on an ST-MRF image segmentation technology, and extracting a container class; using a DINO-X target detection model to identify whether a container cover exists or not, and marking a container without a cover as'no goods, to be loaded and unloaded '; and through a multi-feature fusion loading and unloading state probability model MFSM, the container cover coverage rate, the cargo height and the inclination angle are integrated to judge the state of the container. The accuracy and efficiency of container loading and unloading state recognition are improved, and the intelligent level of port container loading and unloading management is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of port container cargo loading and unloading management, and in particular to a port container cargo loading and unloading status recognition method based on computer vision. Background Art

[0002] With the rapid development of the modern logistics industry, ports, as key nodes in international logistics, are becoming increasingly important for efficient containerized cargo loading and unloading management. Traditional methods for managing containerized cargo loading and unloading rely primarily on manual observation and operation, which not only consumes significant human resources but also struggles to ensure efficiency and accuracy. Furthermore, given the complex port environment and the frequent loading and unloading of containerized cargo, existing technologies face technical bottlenecks in real-time identification and classification, making them unable to meet the demands of modern port management.

[0003] The current methods for container cargo loading and unloading identification mainly include the following: 1) Identification methods based on visual image processing: This type of method identifies containers through static or dynamic images, and combines machine learning or deep learning algorithms for classification. Although it can improve the recognition accuracy to a certain extent, the accuracy is affected in complex lighting, haze and other environments. 2) Sensor-based identification methods: Sensors (such as lidar, ultrasound, etc.) are used to identify the distance and shape of containers. Although it can improve the stability of recognition under certain conditions, it has poor adaptability to multi-target recognition and dynamic environments. 3) Identification methods based on drones and multispectral technology: In recent years, drones have combined multispectral data of visible light and infrared images to monitor the loading and unloading status of containers. They can obtain more comprehensive information under different lighting and climatic conditions, but this method still faces technical difficulties in data fusion and image processing.

[0004] While these methods have improved recognition efficiency to a certain extent, they still face the following technical challenges: 1) Environmental complexity: Port environments are often affected by lighting variations, smoke, shadows, and other factors, which can degrade image quality and affect recognition accuracy. 2) High real-time requirements: Frequent container loading and unloading activities at ports require real-time and accurate state recognition, and existing methods cannot meet the needs of efficient port operations. 3) Target overlap and occlusion: Overlapping and partial occlusion of containers and other objects can reduce the recognition accuracy of target detection models, leaving significant room for improvement in classification accuracy.

[0005] By improving existing technologies, especially introducing more advanced target detection, image segmentation and data fusion technologies, the above problems can be better solved and the efficiency and accuracy of container loading and unloading status recognition can be improved. Summary of the Invention

[0006] Purpose of the invention: The purpose of the present invention is to provide a method for identifying the loading and unloading status of container cargo in ports based on computer vision, which can realize the intelligent identification, classification and management of container loading and unloading activities through efficient models and algorithms, improve port transportation efficiency, reduce the burden of manual operation, and ensure data accuracy.

[0007] The technical solution of the present invention is: a method for identifying the loading and unloading status of a port container cargo based on computer vision, comprising the following steps:

[0008] (1) Using a multispectral drone to collect visible light and infrared video images of the port, and preprocessing the images with a dehazing algorithm;

[0009] (2) Identify and classify port objects in the video based on the improved EfficientDet target detection model, and dynamically adjust the feature fusion weights;

[0010] (3) Combined with motion detection technology, if the detected target is moving, it is classified as a "truck" class; otherwise, it is classified as a "container" class;

[0011] (4) Based on the ST-MRF image segmentation technology, the detection target is further optimized and classified to extract the container class;

[0012] (5) Use the DINO-X target detection model to identify whether the container has a lid. Containers without lids are marked as "no cargo, waiting to be loaded and unloaded." Containers with lids proceed to step (6).

[0013] (6) The container status is judged by combining the cover coverage, cargo height and tilt angle through the multi-feature fusion loading and unloading status probability model MFSM.

[0014] Furthermore, the implementation process of step (1) is as follows:

[0015] A method based on dark color prior is used to remove the dark channel components in the image and restore the image clarity:

[0016] J(x)=I(x)×t(x)+A×(1-t(x))(1)

[0017] Where J(x) represents the dehazed image; I(x) represents the original image collected by the multispectral drone; A represents the air light impact factor, and the brightest 0.1% pixel area in the image is selected and the maximum GRB value is calculated as the air light value; x represents the pixel in the image; t(x) is the transmittance image, which represents the proportion of light passing through the haze, and is calculated as follows:

[0018]

[0019] Where ω represents the dehazing intensity coefficient of the image; Ω(x) is used as the recognition window area to calculate the dark channel value; R(y), G(y), and B(y) are the values of the pixel y in the RGB channels.

[0020] Further combined with infrared images, multispectral fusion is performed; information from different wavelengths is used to enhance the identifiability of the target. A multispectral data melting method based on weighted average is used to fuse the values of each pixel in the visible light image and the infrared image. The fusion ratio is determined based on the weighting coefficient to adjust the proportion of information from different wavelengths:

[0021] I fusion (x) = w visible I visible (x)+w infrared I infrared (x) (3)

[0022] Where, I fusion (x) is the pixel value of the fused image; I visible (x) is the pixel value of the visible light image after defogging; I infrared (x) is the pixel value of the corresponding position in the infrared image; w visible is the weight of the visible light image; w infrared is the weight of the infrared image; weight w visible and w infrared It can be dynamically adjusted according to the actual lighting conditions and haze level of the environment. In the case of heavy haze, the weight of the infrared image can be increased.

[0023] Furthermore, the implementation process of step (2) is as follows:

[0024] Using EfficientNet as the backbone network reduces computational complexity while improving detection accuracy. Furthermore, through the efficient bidirectional feature pyramid network (BiFPN) feature fusion method, multi-scale target detection capabilities are enhanced, allowing feature fusion weights to be dynamically adjusted based on the detection environment.

[0025] Each frame of image is input into the EfficientDet model, and the input image I is processed through a series of convolutional layers. Each convolutional layer extracts image features at different levels and transfers them to the next layer for further processing to generate a feature map F. i :

[0026] F i =Conv(F i-1 ,W i 0 (4)

[0027] In the formula, Conv represents the convolution transformation operation, F i-1 is the feature map of the previous layer, Wi is the convolution kernel of this layer; in order to ensure the effective combination of global information and detail information, bidirectional feature fusion BiFPN is used to fuse feature maps of different scales:

[0028] F i ′=Upsample(F i )+F i+1 (5)

[0029] Among them, Upsample(F i ) is an upsampling operation to improve the image resolution; through the upsampling operation and the higher-level feature map F k+1 Fusion is performed to obtain a new feature map F i ′; Feature fusion weight w i Adopt Softmax dynamic adjustment mechanism and adjust through trainable parameters:

[0030]

[0031] Where, α i It is a trainable parameter of feature layer i. The model will automatically learn appropriate weights according to the scenario. Softmax normalization is also used to ensure that the sum of all weights is 1.

[0032] EfficientDet uses bounding box regression to predict the position of the object (x', y', w', h'0, which represents the center coordinates and width and height of each target's predicted bounding box; and then trains the model through regression loss, where (x, y, w, h) is the initial center coordinates and width and height of the target:

[0033]

[0034] For each predicted box, detect the object category in the box: use the target classification method and output the probability values of c categories through a fully connected layer:

[0035]

[0036] Where, f c is the score corresponding to category c, and p(c) is the probability of that category;

[0037] For the SE module, recalculate the channel weights:

[0038] s=σ(W2·ReLU(W1·z)) (9)

[0039] Among them, W1 and W2 are the weight matrices of the two-layer fully connected layer network; σ represents the Sigmoid activation function; channel weighting: s c is the channel weight, which is used to adjust the importance of the feature; Fc ′ is the feature after attention weighting:

[0040] F c ′=s c ·F c (10)

[0041] Use non-maximum suppression Soft-NMS to reduce the confidence of adjacent targets:

[0042]

[0043] Among them, S i is the confidence score of the i-th detection box; IoU(i,j) represents the intersection-over-union ratio of the i-th box and the j-th box; σ controls the degree of confidence decay;

[0044] After obtaining the bounding box coordinates and category probabilities of all candidate boxes, the non-maximum suppression algorithm is applied to remove duplicate boxes and the optimal box is selected by calculating the overlap IoU between the candidate boxes:

[0045]

[0046] All candidate boxes are sorted according to their probability, the box with the highest score is selected, and then other boxes whose IoU with the selected box exceeds the set threshold are gradually removed.

[0047] Furthermore, the implementation process of step (3) is as follows:

[0048] The data feature layer is gray-scale processed to form a continuous gray-scale image frame F t ,F t-1 ,F t-2 , the frame difference method is used to calculate the frame difference image between adjacent frames:

[0049] |F t (x,y)-F t-1 (x,y| (13)

[0050] |F t-1 (x,y)-F t-2 (x,y)| (14)

[0051] Among them, F t (x, y) is the grayscale image frame at time t, which represents the grayscale value of the pixel point (x, y) at time t; F t-1 (x, y) is the grayscale image frame at time t-1; F t-2 (x, y) is the grayscale image frame at time t-2; Equation (13) is the change amplitude of the pixel point (x, y) between the current frame and the previous frame, that is, the motion program; Equation (14) is the change amplitude of the pixel point (x, y) between the previous frame and the previous two frames;

[0052] Set the grayscale threshold G t , extract the pixel gray value greater than G in the frame difference image t A set of pixels:

[0053] N={(x,y)∈R||F t (x,y)-F t-1 (x,y)|≥G t ∨|F t-1 (x,y)-F t-2 (x,y)|≥G t} (15)

[0054] Where N is the set of pixels that meet the motion condition, that is, the set of pixels whose change amplitude exceeds the threshold; R is the detection area, that is, a region of interest ROI in the image; G t It is the grayscale threshold, which is used to determine whether the pixel belongs to the motion area. t , then the pixel is considered to belong to the motion area;

[0055] The extracted pixel set N is used as the detection area R, and the motion density of the detection area R is calculated:

[0056]

[0057] Where S(N) is the number of pixels in the set N, that is, the number of pixels detected as moving; S(R) is the number of all pixels in the detection area R; according to the motion density of the detection area R, that is, the proportion of moving pixels d f (R) and the set motion density threshold D f Determine the motion state V m (R):

[0058]

[0059] If V m (R) = true, it means that the vehicle in the detection area is moving and is classified as a "truck"; if V m (R) = false, which means that there is no movement of the vehicle in the detection area, and it is classified as a "container" class.

[0060] Furthermore, the implementation process of step (4) is as follows:

[0061] ST-MRF introduces the temporal dimension and combines the spatiotemporal information of multiple frames in the image sequence to improve the stability and accuracy of segmentation in dynamic videos. ST-MRF achieves image segmentation by minimizing the total energy function E(L):

[0062] E(L)=E data (L)+λ1E smooth (L)+λ2E temporal (L) (18)

[0063] Among them, L is the pixel label set to be optimized; E data (L) is a data item used to describe the matching degree between pixel features and their labels; E smooth (L) acts as a spatial smoothing term, encouraging adjacent pixels to have the same labels to avoid spurious noise in the segmented image; E temporal (L) is a temporal consistency term that encourages the labels of the same pixel in adjacent frames to remain consistent, reducing flickering in the video; λ1 and λ2 are balancing factors that adjust the contribution of spatial smoothness and temporal consistency to the total energy;

[0064] Data item E data (L) is:

[0065]

[0066] Where P represents the set of pixels in the image; L p is the segmentation label of pixel p; D p (L p ) is the feature cost of pixel p;

[0067] Spatial smoothing term E smooth (L) is:

[0068]

[0069] Where N is the set of pixel adjacency pairs; V pq (L p ,L q ) If p and q are adjacent pixels, and L p ≠L q Then a penalty is generated, and γ is used as the spatial smoothing penalty coefficient to control the degree of smoothing;

[0070] Time consistency term E temporal (L) is:

[0071]

[0072] in, represents the label of the same pixel p in the previous frame; If the pixel labels of the current frame are inconsistent with those of the previous frame, a penalty is imposed; the total energy function E(L) is minimized through the image optimization algorithm to obtain the segmented pixel label L, which is the final container segmentation area.

[0073] Furthermore, the implementation process of step (5) is as follows:

[0074] DINO-X uses a convolutional neural network to extract feature maps from input images. The image is processed layer by layer through multiple convolutional layers, pooling layers, and activation functions, ultimately outputting a series of high-level feature maps:

[0075] F=CNN(I)(23)

[0076] Where I is the input image; F is the extracted feature map;

[0077] Further use bounding box regression and classification tasks to identify the state of the box lid: Bounding box regression predicts the position and size of the box lid:

[0078] BBox cover =[x min ,y min ,x max ,y max ] (twenty four)

[0079] Among them, x min ,y min is the coordinate of the upper left corner of the box cover; x max ,y max is the coordinate of the lower right corner of the lid; if the confidence of the predicted lid position is high and the model believes there is an object at that position, the container is considered to have a lid; if the confidence is low or no obvious object is detected, the container is considered to have no lid:

[0080] P cover =Sigmoid(f(F))(25)

[0081] Among them, P cover is the probability of the box lid existing; f(F) is the value obtained by the classifier after the feature map F processed by the neural network, and the probability output by the Sigmoid activation function: according to P cover The value of determines the state of the box cover. If P cover >0.5, it is judged as a "covered container". If P cover If ≤0.5, it is judged as an "uncovered container";

[0082] For a container without a lid, if the DINO-X model determines that the lid does not exist, that is, P cover ≤0.5, the container is marked as “no cargo, waiting to be loaded and unloaded”; if the DINO-X model determines that the lid exists, that is, P cover >0.5, the container enters the loading and unloading state:

[0083]

[0084] Furthermore, the implementation process of step (6) is as follows:

[0085] Cover coverage:

[0086]

[0087] Among them, C cover A represents the box cover coverage rate, which indicates the proportion of the box cover area; cover is the area of the area covered by the box lid; A container is the total area of the container; confidence level P>0.9 means “loaded and unloaded”, 0.5<P≤0.9 means “there is cargo, loading and unloading in progress”, and P≤0.5 means “there is no cargo, waiting to be loaded and unloaded”;

[0088] Cargo height: Cargo height H cargo It is the height of the cargo in the container to determine whether the container is full or whether some cargo is still being unloaded:

[0089] H cargo =max(H current *H base ) (28)

[0090] Among them, H cargo Indicates the height of the cargo in the container; H current Expresses the highest point of the current goods; H base Indicates the base height of the container bottom. Cargo height greater than the preset value may indicate "loaded, loading or unloading" or "loaded or unloaded". Cargo height less than or equal to the preset value indicates that the container is not fully loaded or is in the process of unloading.

[0091] Tilt angle: The tilt angle θ refers to the tilt of the container, indicating the tilt angle of the container relative to the ground. The tilt angle is used to determine whether the container is in a normal loading and unloading position, or is tilted due to improper operation:

[0092]

[0093] Where θ is the tilt angle, representing the tilt angle of the container; Δy is the vertical offset of the container; and Δx is the horizontal offset of the container. A tilt angle greater than the preset value indicates that the container is unstable and there may be problems during loading and unloading. Under normal circumstances, when the tilt angle is close to 0 degrees, it means that the container may be "loaded or unloaded" or "loaded and unloaded".

[0094] The above three features are weighted and integrated to obtain the final probability P of the loading and unloading status. The influence of each feature is weighted with a weight W1, W2, W3 to finally obtain the comprehensive probability P of the container loading and unloading status:

[0095] P=W1·C cover +W2·H cargo +W3·θ (30)

[0096] Among them, P refers to the comprehensive probability of container loading and unloading status; C cover Refers to the box cover coverage rate; H cargo refers to the height of the cargo; θ refers to the tilt angle; W1, W2, and W3 are the weights of each feature, which determine the importance of each feature in the final judgment; based on the value of the comprehensive probability P, the status of the container is divided into the following:

[0097]

[0098] Beneficial effects: Compared with the existing technology, the beneficial effects of the present invention are: the present invention can improve the recognition efficiency of container cargo loading and unloading, reduce the computing load, and achieve high-precision real-time detection; the present invention optimizes the transportation process of the port, reduces manual intervention, and improves the level of port automation management. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] Figure 1 is a flow chart of the present invention;

[0100] Figure 2 A schematic diagram of the model of the present invention for identifying, classifying and labeling port objects;

[0101] Figure 3 This is an example diagram of a container in the loading and unloading state of the present invention being a category to be loaded and unloaded;

[0102] Figure 4 This is an example diagram of the container loading and unloading status of the present invention being completed. DETAILED DESCRIPTION

[0103] The present invention will be described in further detail below with reference to the accompanying drawings.

[0104] like Figure 1 As shown, the present invention proposes a method for identifying the loading and unloading status of port container cargo based on computer vision, which specifically includes the following steps:

[0105] Step 1: Video data acquisition and processing: Visible light and infrared video images of the port are collected using a multispectral drone, and the images are pre-processed using a dehazing algorithm.

[0106] Under the influence of different environmental factors such as lighting and smoke, the image quality is degraded, making it difficult to accurately detect and judge the target state. Therefore, this study uses a multispectral drone to collect images of port objects, including visible light images and infrared images. In order to improve the image quality and reduce the impact of haze or low-light environments on target recognition, a method based on dark channel prior (DCP) is used to remove the dark channel components in the image and restore the image clarity.

[0107] J(x)=I(x)×t(x)+A×(1-t(x))(1)

[0108] Where J(x) represents the dehazed image; I(x) represents the original image collected by the multispectral drone; A represents the air light impact factor, and the brightest 0.1% pixel area in the image is selected and the maximum GRB value of this area is calculated as the air light value; x represents the pixel point in the image; t(x) is the transmittance image, which represents the proportion of light passing through the haze, and is calculated as follows:

[0109]

[0110] Where ω represents the dehazing strength coefficient of the image, which is usually set to 0.95; Ω(x) is used as the recognition window area to calculate the dark channel value; R(y), G(y), and B(y) are the values of the pixel y in the RGB channels.

[0111] The image processed by the dehazing algorithm may still be affected by factors such as lighting changes, shadows, and reflections. Therefore, it is further combined with infrared images and multispectral fusion is performed to obtain more comprehensive image information.

[0112] The information of different wavelengths is used to enhance the identifiability of the target. A multispectral data melting method based on weighted average is used to fuse the values of each pixel in the visible light image and the infrared image. The fusion ratio is determined based on the weighting coefficient to adjust the proportion of information from different wavelengths:

[0113] I fusion (x) = w visible I visible (x)+w infrared I infrared (x)(3)

[0114] Where, I fusion (x) is the pixel value of the fused image; I visible (x) is the pixel value of the visible light image after defogging; I infrared (x) is the pixel value of the corresponding position in the infrared image; w visible is the weight of the visible light image; winfrared is the weight of the infrared image; weight w visible and w infrared It can be dynamically adjusted according to the actual lighting conditions and haze level of the environment. In the case of heavy haze, the weight of the infrared image can be increased.

[0115] Step 2: Object recognition and classification: Identify and classify port objects in the video based on the improved EfficientDet target detection model, and dynamically adjust the feature fusion weights.

[0116] This paper adopts an improved EfficientDet target detection model: using EfficientNet as the backbone network, it reduces the amount of computation while improving detection accuracy; at the same time, through the efficient bidirectional feature pyramid network (BiFPN) feature fusion method, it improves the multi-scale target detection capability, so that the feature fusion weights can be dynamically adjusted according to the detection environment.

[0117] Each frame of image is input into the EfficientDet model, and the input image I is processed through a series of convolutional layers. Each convolutional layer extracts image features at different levels and transfers them to the next layer for further processing to generate a feature map F. i :

[0118] F i =Conv(F i-1 ,W i )(4)

[0119] In the formula, Conv represents the convolution transformation operation, F i-1 is the feature map of the previous layer, W i is the convolution kernel of this layer; in order to ensure the effective combination of global information and detail information, bidirectional feature fusion (BiFPN) is used to fuse feature maps of different scales:

[0120] F i ′=Upsample(F i )+F i+1 (5)

[0121] Among them, Upsample(F i ) is an upsampling operation to improve the image resolution; through the upsampling operation and the higher-level feature map F k+1 Fusion is performed to obtain a new feature map F i ′; Feature fusion weight w i Adopt Softmax dynamic adjustment mechanism and adjust through trainable parameters:

[0122]

[0123] Where, α i It is a trainable parameter of feature layer i. The model will automatically learn appropriate weights according to the scenario. At the same time, Softmax normalization is used to ensure that the sum of all weights is 1.

[0124] EfficientDet uses bounding box regression to predict the position of the object (x', y', w', h'), which represents the center coordinates and width and height of each target's predicted bounding box; then the model is trained through regression loss, where (x, y, w, h) are the initial center coordinates and width and height of the target:

[0125]

[0126] For each predicted box, detect the object category in the box: use the target classification method and output the probability values of c categories through a fully connected layer:

[0127]

[0128] Where, f c is the score corresponding to category c, and p(c) is the probability of that category

[0129] The present invention recalculates the channel weights of the SE (Squeeze-and-Excitation) module:

[0130] s=σ(W2·ReLU(W1·z)) (9)

[0131] W1 and W2 are the weight matrices of the two fully connected layer networks; σ represents the Sigmoid activation function; channel weighting: s c is the channel weight (between 0 and 1), which is used to adjust the importance of the feature; F c ′ is the feature after attention weighting. The improved model improves the detection accuracy, reduces background interference, and enables the model to focus on key targets (such as containers and trucks).

[0132] F c ′=s c ·F c (10)

[0133] Since there may be a large number of overlapping containers in the port scene, the model may mistakenly delete targets. Therefore, Soft-NMS (non-maximum suppression) is used to reduce the confidence of adjacent targets:

[0134]

[0135] Among them, S iis the confidence score of the i-th detection box; IoU(i,j) represents the intersection over union ratio of the i-th box and the j-th box; σ controls the degree of confidence decay.

[0136] After obtaining the bounding box coordinates and category probabilities of all candidate boxes, the non-maximum suppression algorithm is applied to remove duplicate boxes and the optimal box is selected by calculating the overlap (IoU) between the candidate boxes:

[0137]

[0138] All candidate boxes are sorted according to their probability, the box with the highest score is selected, and then other boxes whose IoU with the selected box exceeds the set threshold are gradually removed.

[0139] Step 3: Motion detection and classification: Combined with motion detection technology, if the detected target is moving, it is classified as a "truck" class; otherwise, it is classified as a "container" class.

[0140] The data feature layer is gray-scale processed to form a continuous gray-scale image frame F t ,F t-1 ,F t-2 , the frame difference method is used to calculate the frame difference image between adjacent frames:

[0141] |F t (x,y)-F t-1 (x,y)| (13)

[0142] |F t-1 (x,y)-F t-2 (x,y)| (14)

[0143] F t (x, y) is the grayscale image frame at time t, which represents the grayscale value of the pixel point (x, y) at time t; F t-1 (x, y) is the grayscale image frame at time t-1; F t-2 (x, y) is the grayscale image frame at time t-2; Equation (13) is the change amplitude of the pixel point (x, y) between the current frame and the previous frame, that is, the motion program; Equation (14) is the change amplitude of the pixel point (x, y) between the previous frame and the previous two frames.

[0144] Set the grayscale threshold G t , extract the pixel gray value greater than G in the frame difference image t A set of pixels:

[0145] N={(x,y)∈R||F t (x,y)-F t-1 (x,y)|≥G t ∨|F t-1 (x,y)-Ft-2 (x,y)|≥G t} (15)

[0146] Where N is the set of pixels that meet the motion condition, that is, the set of pixels whose change amplitude exceeds the threshold; R is the detection area, that is, a region of interest (ROI) in the image; G t It is the grayscale threshold, which is used to determine whether the pixel belongs to the motion area. t , then the pixel is considered to belong to the motion area.

[0147] The extracted pixel set N is used as the detection area R, and the motion density of the detection area R is calculated:

[0148]

[0149] Where S(N) is the number of pixels in the set N, that is, the number of pixels detected as moving; S(R) is the number of all pixels in the detection area R; according to the motion density (that is, the proportion of moving pixels) d f (R) and the set motion density threshold D f Determine the motion state V m (R):

[0150]

[0151] If V m (R) = true, it means that the vehicle in the detection area is moving and is classified as a "truck"; if V m (R) = false, which means that there is no movement of the vehicle in the detection area, and it is classified as a "container" class.

[0152] Step 4: Optimize and classify the detection targets using image segmentation technology. Further optimize and classify the detection targets based on ST-MRF image segmentation technology to extract the container class.

[0153] The use of ST-MRF introduces the temporal dimension (Temporal), combines the spatiotemporal information of multiple frames in the image sequence, and improves the stability and accuracy of segmentation in dynamic videos. ST-MRF achieves image segmentation by minimizing the following energy function: E(L) is the total energy, and the goal is to minimize this value to obtain the optimal segmentation result; L is the set of pixel labels to be optimized (such as "container" or "non-container"); E data (L) is used as a data item to describe the matching degree between pixel features and their labels; E smooth (L) acts as a spatial smoothing term, encouraging adjacent pixels to have the same labels to avoid spurious noise in the segmented image; E temporal(L) is a temporal consistency term that encourages the labels of the same pixel in adjacent frames to remain consistent, reducing the flickering problem in the video; λ1 and λ2 are balancing factors that adjust the contribution of spatial smoothness and temporal consistency to the total energy:

[0154] E(L)=E data (L)+λ1E smooth (L)+λ2E temporal (L) (18)

[0155] Express each energy term specifically:

[0156] 1) Data item E data (L): P represents the set of pixels in the image; L p is the segmentation label of pixel p (1 for container, 0 for background); D p (L p ) is the feature cost of pixel p, which is usually based on grayscale, texture, color and other features to describe the degree of match between the pixel and the label.

[0157]

[0158] 2) Spatial smoothing term E smooth (L): N is the set of adjacent pairs of pixels; V pq (L p ,L q ) If p and q are adjacent pixels, and L p ≠L q Then a penalty is generated, γ is used as the spatial smoothing penalty coefficient to control the degree of smoothing; the usual form is:

[0159]

[0160] 3) Time consistency term E temporal (L): represents the label of the same pixel p in the previous frame; If the pixel labels of the current frame are inconsistent with those of the previous frame, a penalty is imposed (Formula 22):

[0161]

[0162] This energy function is minimized using image optimization algorithms such as Graph Cut or Loopy Belief Propagation to obtain the segmented pixel labels L, which are the final container segmentation regions. The optimal label configuration is obtained by optimizing the total energy function E(L).

[0163] Step 5: Determine the condition of the container cover: Use the DINO-X target detection model to identify whether the container cover is present. Containers without covers are marked as "no cargo, waiting to be loaded and unloaded." Containers with covers proceed to step 6.

[0164] The DINO-X object detection model is used to identify whether the container lid is covered or not, and classify it into two categories: "without lid" and "with lid".

[0165] DINO-X uses a convolutional neural network to extract feature maps from input images. The image is processed layer by layer through multiple convolutional layers, pooling layers, activation functions, and other network layers, ultimately outputting a series of high-level feature maps:

[0166] F=CNN(I)(23)

[0167] Where I is the input image (container image); F is the extracted feature map, which includes important information in the image, such as the shape, color, and lid status of the container.

[0168] For each container, the model needs to determine whether there is a lid. This judgment is made by further analyzing the feature map and using bounding box regression and classification tasks to identify the status of the lid: Bounding Box Regression (Bounding Box Regression) predicts the position and size of the lid:

[0169] BBox cover =[x min ,y min ,x max ,y max ] (twenty four)

[0170] Among them, x min ,y min is the coordinate of the upper left corner of the box cover; x max ,y max is the coordinate of the lower right corner of the container lid; based on the analysis of the feature graph, the model determines whether the container lid exists. If the confidence of the predicted container lid position is high and the model believes there is an object at that location, the container is considered to have a lid; if the confidence is low or no obvious object is detected, the container is considered to have no lid:

[0171] P cover =Sigmoid(f(F))(25)

[0172] P cover is the probability of the box lid existing; f(F) is the value obtained by the classifier after the feature map F processed by the neural network, and the probability output by the Sigmoid activation function: according to P cover The value of determines the state of the box cover. If P cover >0.5, it is judged as a "covered container". If Pcover If ≤ 0.5, it is determined as an "uncovered container", and the specific recognition result is as Figure 2 shown.

[0173] For an uncovered container: If the DINO-X model determines that the lid does not exist (i.e., P cover ≤ 0.5), then the container is marked as "no goods, waiting for loading and unloading"; if the DINO-X model determines that the lid exists (i.e., P cover > 0.5), then the container enters the next judgment for the loading and unloading state:

[0174]

[0175] Step 6: Judgment of container loading and unloading situation: Through the multi-feature fusion loading and unloading state probability model (MFSM), comprehensively judge the container state based on the lid coverage rate, cargo height, and tilt angle: The confidence level P > 0.9 is "loaded and unloaded", 0.5 < P ≤ 0.9 is "there are goods, loading and unloading in progress", and P ≤ 0.5 is "no goods, waiting for loading and unloading".

[0176] 1) Lid coverage rate: The confidence level P > 0.9 is "loaded and unloaded", 0.5 < P ≤ 0.9 is "there are goods, loading and unloading in progress", and P ≤ 0.5 is "no goods, waiting for loading and unloading":

[0177]

[0178] C cover represents the lid coverage rate, indicating the proportion of the lid-covered area; A cover is the area of the lid-blocked area; A container is the total area of the container; when the lid coverage rate is high, it indicates that the lid is intact and may be "loaded and unloaded" or "there are goods, loading and unloading in progress"; when the lid coverage rate is low, it may indicate that the container has been partially opened or has no lid and may be "no goods, waiting for loading and unloading".

[0179] 2) Cargo height: The cargo height H cargo is the height of the goods in the container to judge whether the container is full or whether there is still some goods being unloaded:

[0180] H cargo = max(H current - H base ) (28)

[0181] Among them, H cargo represents the height of the goods in the container; H current expresses the highest point of the current goods; H baseIndicates the base height of the container bottom. A higher cargo height usually means that the container has a lot of cargo in it, which may be "Loading or unloading" or "Loading or unloading completed". A lower cargo height may mean that the container is not completely full or is in the process of being unloaded.

[0182] 3) Tilt angle: The tilt angle θ refers to the tilt of the container, which indicates the tilt angle of the container relative to the ground. The tilt angle is used to determine whether the container is in a normal loading and unloading position, or is tilted due to improper operation:

[0183]

[0184] Where θ is the tilt angle, representing the tilt angle of the container; Δy is the height offset of the container; and Δx is the horizontal offset of the container. A container with a large tilt angle may mean that the container is unstable and there may be problems during loading and unloading. Under normal circumstances, when the tilt angle is close to 0 degrees, the container may be "loaded or unloaded" or "with cargo, loading or unloading in progress."

[0185] The above three features are weighted and integrated to obtain the final probability P of loading and unloading status: the influence of each feature is weighted with a weight W1, W2, W3, and finally the comprehensive probability P of container loading and unloading status is obtained:

[0186] P=W1·C cover +W2·H cargo +W3·θ (30)

[0187] Among them, P refers to the comprehensive probability of container loading and unloading status; C cover Refers to the box cover coverage rate; H cargo Refers to the height of the cargo; θ refers to the tilt angle; W1, W2, and W3 are the weights of each feature, which determine the importance of each feature in the final judgment. According to the value of the comprehensive probability P, the status of the container can be divided into the following categories. The specific recognition results are as follows: Figure 3 、 Figure 4 As shown:

[0188]

[0189] The above embodiments are intended only to illustrate the technical concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. They are not intended to limit the scope of protection of the present invention. Any equivalent changes or modifications made in accordance with the spirit of the present invention are intended to be covered by the scope of protection of the present invention.

Claims

1. A method for identifying the loading and unloading status of port container cargo based on computer vision, characterized in that: The following steps are involved: (1) Using a multispectral drone to collect visible light and infrared video images of the port, and preprocessing the images with a dehazing algorithm; (2) Identify and classify port objects in the video based on the improved EfficientDet target detection model, and dynamically adjust the feature fusion weights; (3) Combined with motion detection technology, if the detected target is moving, it is classified as a "truck" class; otherwise, it is classified as a "container" class; (4) Based on the ST-MRF image segmentation technology, the detection target is further optimized and classified to extract the container class; (5) Use the DINO-X target detection model to identify whether the container has a lid. Containers without lids are marked as "no cargo, waiting to be loaded and unloaded." Containers with lids proceed to step (6). (6) The container status is judged by combining the cover coverage, cargo height and tilt angle through the multi-feature fusion loading and unloading status probability model MFSM.

2. The method for identifying the loading and unloading status of a port container cargo based on computer vision according to claim 1, characterized in that: The implementation process of step (1) is as follows: A method based on dark color prior is used to remove the dark channel components in the image and restore the image clarity: J(x)=I(x)×t(x)+A×(1-t(x))(1) Where J(x) represents the dehazed image; I(x) represents the original image collected by the multispectral drone; A represents the air light impact factor, and the brightest 0.1% pixel area in the image is selected and the maximum GRB value is calculated as the air light value; x represents the pixel in the image; t(x) is the transmittance image, which represents the proportion of light passing through the haze, and is calculated as follows: Where ω represents the dehazing intensity coefficient of the image; Ω(x) is used as the recognition window area to calculate the dark channel value; R(y), G(y), and B(y) are the values of the pixel y in the RGB channels. Further combined with infrared images, multispectral fusion is performed; information from different wavelengths is used to enhance the identifiability of the target. A multispectral data melting method based on weighted average is used to fuse the values of each pixel in the visible light image and the infrared image. The fusion ratio is determined based on the weighting coefficient to adjust the proportion of information from different wavelengths: I fusion (x)=w visible ·I visible (x)+w infrared ·I infrared (x) (3) Where, I fusion (x) is the pixel value of the fused image; I visible (x) is the pixel value of the visible light image after defogging; I infrared (x) is the pixel value of the corresponding position in the infrared image; w visible is the weight of the visible light image; w infrared is the weight of the infrared image; weight w visible and w infrared It can be dynamically adjusted according to the actual lighting conditions and haze level of the environment. In the case of heavy haze, the weight of the infrared image can be increased.

3. The method for identifying the loading and unloading status of a port container cargo based on computer vision according to claim 1, characterized in that: The implementation process of step (2) is as follows: Using EfficientNet as the backbone network reduces computational complexity while improving detection accuracy. Furthermore, through the efficient bidirectional feature pyramid network (BiFPN) feature fusion method, multi-scale target detection capabilities are enhanced, allowing feature fusion weights to be dynamically adjusted based on the detection environment. Each frame of image is input into the EfficientDet model, and the input image I is processed through a series of convolutional layers. Each convolutional layer extracts image features at different levels and transfers them to the next layer for further processing to generate a feature map F. i : F i =Conv(F i-1 ,W i ) (4) In the formula, Conv represents the convolution transformation operation, F i-1 is the feature map of the previous layer, W i is the convolution kernel of this layer; in order to ensure the effective combination of global information and detail information, bidirectional feature fusion BiFPN is used to fuse feature maps of different scales: F i '=Upsample(F i )+F i+1 (5) Among them, Upsample(F i ) is an upsampling operation to improve the image resolution; through the upsampling operation and the higher-level feature map F k+1 Fusion is performed to obtain a new feature map F i '; Feature fusion weight w i Adopt Softmax dynamic adjustment mechanism and adjust through trainable parameters: Where, α i It is a trainable parameter of feature layer i. The model will automatically learn appropriate weights according to the scenario. Softmax normalization is also used to ensure that the sum of all weights is 1. EfficientDet uses bounding box regression to predict the position of the object (x', y', w', h'0, which represents the center coordinates and width and height of each target's predicted bounding box; and then trains the model through regression loss, where (x, y, w, h) is the initial center coordinates and width and height of the target: For each predicted box, detect the object category in the box: use the target classification method and output the probability values of c categories through a fully connected layer: Where, f c is the score corresponding to category c, and p(c) is the probability of that category; For the SE module, recalculate the channel weights: s=σ(W2·ReLU(W1·z)) (9) Among them, W1 and W2 are the weight matrices of the two-layer fully connected layer network; σ represents the Sigmoid activation function; channel weighting: s c is the channel weight, which is used to adjust the importance of the feature; F c ′ is the feature after attention weighting: F c ′=s c ·F c (10) Use non-maximum suppression Soft-NMS to reduce the confidence of adjacent targets: Among them, S i is the confidence score of the i-th detection box; IoU(i,j) represents the intersection-over-union ratio of the i-th box and the j-th box; σ controls the degree of confidence decay; After obtaining the bounding box coordinates and category probabilities of all candidate boxes, the non-maximum suppression algorithm is applied to remove duplicate boxes and the optimal box is selected by calculating the overlap IoU between the candidate boxes: All candidate boxes are sorted according to their probability, the box with the highest score is selected, and then other boxes whose IoU with the selected box exceeds the set threshold are gradually removed.

4. The method for identifying the loading and unloading status of a port container cargo based on computer vision according to claim 1, characterized in that: The implementation process of step (3) is as follows: The data feature layer is gray-scale processed to form a continuous gray-scale image frame F t ,F t-1 ,F t-2 , the frame difference method is used to calculate the frame difference image between adjacent frames: |F t (x,y)-F t-1 (x,y)| (13) |F t-1 (x,y)-F t-2 (x,y)| (14) Among them, F t (x, y) is the grayscale image frame at time t, which represents the grayscale value of the pixel point (x, y) at time t; F t-1 (x, y) is the grayscale image frame at time t-1; F t-2 (x, y) is the grayscale image frame at time t-2; Equation (13) is the change amplitude of the pixel point (x, y) between the current frame and the previous frame, that is, the motion program; Equation (14) is the change amplitude of the pixel point (x, y) between the previous frame and the previous two frames; Set the grayscale threshold G t , extract the pixel gray value greater than G in the frame difference image t A set of pixels: N={(x,y)∈R|F t (x,y)-F t-1 (x,y)|≥G t ∨|F t-1 (x,y)-F t-2 (x,y)|≥G t } (15) Where N is the set of pixels that meet the motion condition, that is, the set of pixels whose change amplitude exceeds the threshold; R is the detection area, that is, a region of interest ROI in the image; G t It is the grayscale threshold, which is used to determine whether the pixel belongs to the motion area. t , then the pixel is considered to belong to the motion area; The extracted pixel set N is used as the detection area R, and the motion density of the detection area R is calculated: Where S(N) is the number of pixels in the set N, that is, the number of pixels detected as moving; S(R) is the number of all pixels in the detection area R; according to the motion density of the detection area R, that is, the proportion of moving pixels d f (R) and the set motion density threshold D f Determine the motion state V m (R): If V m (R) = true, it means that the vehicle in the detection area is moving and is classified as a "truck"; if V m (R) = false, it means that there is no movement of the vehicle in the detection area, and it is classified as a "container" class.

5. The method for identifying the loading and unloading status of a port container cargo based on computer vision according to claim 1, characterized in that: The implementation process of step (4) is as follows: ST-MRF introduces the temporal dimension and combines the spatiotemporal information of multiple frames in the image sequence to improve the stability and accuracy of segmentation in dynamic videos. ST-MRF achieves image segmentation by minimizing the total energy function E(L): E(L)=E data (L)+λ1E smooth (L)+λ2E temporal (L) (18) Among them, L is the pixel label set to be optimized; E data (L) is a data item used to describe the matching degree between pixel features and their labels; E smooth (L) acts as a spatial smoothing term, encouraging adjacent pixels to have the same labels to avoid spurious noise in the segmented image; E temporal (L) is a temporal consistency term that encourages the labels of the same pixel in adjacent frames to remain consistent, reducing flickering in the video; λ1 and λ2 are balancing factors that adjust the contribution of spatial smoothness and temporal consistency to the total energy; Data item E data (L) is: Where P represents the set of pixels in the image; L p is the segmentation label of pixel p; D p (L p ) is the feature cost of pixel p; Spatial smoothing term E smooth (L) is: Where N is the set of pixel adjacency pairs; V pq (L p ,L q ) If p and q are adjacent pixels, and L p ≠L q Then a penalty is generated, and γ is used as the spatial smoothing penalty coefficient to control the degree of smoothing; Time consistency term E temporal (L) is: in, represents the label of the same pixel p in the previous frame; If the pixel labels of the current frame are inconsistent with those of the previous frame, a penalty is imposed; the total energy function E(L) is minimized through the image optimization algorithm to obtain the segmented pixel label L, which is the final container segmentation area.

6. The method for identifying the loading and unloading status of a port container cargo based on computer vision according to claim 1, characterized in that: The implementation process of step (5) is as follows: DINO-X uses a convolutional neural network to extract feature maps from input images. The image is processed layer by layer through multiple convolutional layers, pooling layers, and activation functions, ultimately outputting a series of high-level feature maps: F = CNN(I) (23) where I is the input image; F is the extracted feature map; Further use bounding box regression and classification tasks to identify the state of the box lid: Bounding box regression predicts the position and size of the box lid: BBox cover [x min ,y min , x max ,y max ] (24) Among them, x min ,y min is the coordinate of the upper left corner of the box cover; x max ,y max is the coordinate of the lower right corner of the lid; if the confidence of the predicted lid position is high and the model believes there is an object at that position, the container is considered to have a lid; if the confidence is low or no obvious object is detected, the container is considered to have no lid: P cover =Sigmoid(f(F))(25) Among them, P cover is the probability of the box lid existing; f(F) is the value obtained by the classifier after the feature map F processed by the neural network, and the probability output by the Sigmoid activation function: according to P cover The value of determines the state of the box cover. If P cover >0.5, it is judged as a "covered container". If P cover If ≤0.5, it is judged as "open container"; For a container without a lid, if the DINO-X model determines that the lid does not exist, that is, P cover ≤0.5, the container is marked as "no cargo, waiting to be loaded and unloaded"; if the DINO-X model determines that the lid exists, that is, P cover >0.5, the container enters the loading and unloading state:

7. The method for identifying the loading and unloading status of a port container cargo based on computer vision according to claim 1, characterized in that: The implementation process of step (6) is as follows: Cover coverage: Among them, C cover A represents the box cover coverage rate, which indicates the proportion of the box cover area; cover is the area of the area covered by the box lid; A container is the total area of the container; confidence level P>0.9 means "loaded and unloaded", 0.5<P≤0.9 means "cargo is available and loading and unloading is in progress", and P≤0.5 means "no cargo is available and waiting to be loaded and unloaded"; Cargo height: Cargo height H cargo It is the height of the cargo in the container to determine whether the container is full or whether some cargo is still being unloaded: H cargo =max(H current -H base )(28) Among them, H cargo Indicates the height of the cargo in the container; H current Expresses the highest point of the current goods; H base Indicates the base height of the container bottom. Cargo height greater than the preset value may indicate "loaded" or "loaded and unloaded." Cargo height less than or equal to the preset value indicates that the container is not fully loaded or is in the process of unloading. Tilt angle: The tilt angle θ refers to the tilt of the container, indicating the tilt angle of the container relative to the ground. The tilt angle is used to determine whether the container is in a normal loading and unloading position, or is tilted due to improper operation: Where θ is the tilt angle, representing the container's tilt; Δy is the container's vertical offset; and Δx is the container's horizontal offset. A tilt angle greater than the preset value indicates that the container is unstable and may have experienced a problem during loading or unloading. Under normal circumstances, a tilt angle close to 0 degrees indicates that the container may be "loaded or unloaded" or "loaded and unloaded." The above three features are weighted and integrated to obtain the final probability P of the loading and unloading status. The influence of each feature is weighted with a weight W1, W2, W3 to finally obtain the comprehensive probability P of the container loading and unloading status: P=W1·C cover +W2·H cargo +W3·θ (30) Among them, P refers to the comprehensive probability of container loading and unloading status; C cover Refers to the box cover coverage rate; H cargo refers to the height of the cargo; θ refers to the tilt angle; W1, W2, and W3 are the weights of each feature, which determine the importance of each feature in the final judgment; based on the value of the comprehensive probability P, the status of the container is divided into the following:

Images