A steel logistics tracking method based on a Mamba model and a BEV

Abstract

The present application relates to a kind of steel logistics tracking methods based on Mamba model and BEV, belong to steel logistics tracking technical field, solve the real-time insufficient of existing steel logistics tracking method, precision is not high, robustness is poor and the problem of cost is too high.The method comprises: obtaining the monitoring video that each camera set in operation area is shot, and frame is extracted, obtain the sample frame sequence of each camera at the same time point;Each frame image is detected using Vision Mamba model, and the target detection information of each logistics target is obtained;The target detection information of each logistics target in the image of the same time point is mapped to the same BEV plane, and the BEV image corresponding to each time point is obtained, and BEV frame sequence is formed;Based on the target detection information of each logistics target in each frame BEV image, target tracking is carried out, and the position and state of each logistics target in each frame time point are obtained based on target tracking result.

Description

Technical Field

[0001] This invention relates to the field of steel logistics tracking technology, and in particular to a steel logistics tracking method based on the Mamba model and BEV. Background Technology

[0002] Steel logistics tracking is a crucial aspect of steel production and logistics management. Its purpose is to monitor and manage the dynamic information of key logistics objects such as steel billets and overhead cranes in real time to ensure the efficiency and safety of the production process.

[0003] Traditional steel logistics tracking systems are mostly based on two-dimensional image processing technology, relying on the perspective of a single camera. When dealing with complex scenes, they are easily affected by occlusion, changes in lighting, and perspective deviations, resulting in insufficient accuracy and robustness in target detection and tracking. Furthermore, existing tracking systems primarily focus on the target's location information, lacking the ability to identify the target's state. This prevents the system from providing richer logistics information and meeting the refined needs of production scheduling and safety management. Summary of the Invention

[0004] Based on the above analysis, the present invention aims to provide a steel logistics tracking method based on the Mamba model and BEV, in order to solve the problems of insufficient real-time performance, low accuracy, poor robustness and high cost of existing steel logistics tracking methods.

[0005] The objective of this invention is mainly achieved through the following technical solutions:

[0006] This invention provides a steel logistics tracking method based on the Mamba model and BEV, comprising the following steps:

[0007] Acquire the monitoring video captured by each camera set in the work area, and extract frames at preset time intervals to obtain the sampling frame sequence of each camera at the same time point;

[0008] The trained Vision Mamba model is used to perform target detection on each frame image in each of the sampled frame sequences to obtain target detection information for each logistics target in each frame image; wherein, the logistics targets include steel billets and overhead cranes;

[0009] The target detection information of each logistics target in the images at the same time point in each of the sampling frame sequences is mapped to the same BEV plane to obtain the BEV image corresponding to each time point, forming a BEV frame sequence.

[0010] Based on the target detection information of each logistics target in each frame of the BEV frame sequence, target tracking is performed on each logistics target in each frame of the BEV image, and the position and status of each logistics target in the work area at each frame time point are obtained based on the target tracking results.

[0011] Furthermore, it also includes using a pre-trained ReID model to extract the appearance features of each of the logistics targets in each frame of the image;

[0012] The step of mapping the target detection information of each logistics target in the images at the same time point in each of the sampling frame sequences to the same BEV plane includes:

[0013] Project the bounding boxes of each logistics target in each frame image at the same time point in the sampling frame sequence onto the same BEV plane to obtain the BEV bounding boxes of each logistics target in the BEV plane.

[0014] Based on the BEV bounding boxes of each logistics target and the appearance features extracted from each frame image at the corresponding time point, the logistics targets are deduplicated to obtain the bounding boxes and corresponding appearance features of each logistics target in the BEV plane after deduplication at the corresponding time point.

[0015] Furthermore, obtaining the BEV bounding box of each of the logistics targets on the BEV plane includes: projecting each corner point of the bounding box of each of the logistics targets onto the BEV plane using inverse perspective mapping to obtain the BEV bounding box of each of the logistics targets.

[0016] Furthermore, the process of deduplicating each of the logistics targets to obtain the bounding box and corresponding appearance features of each deduplicated logistics target in the BEV plane includes:

[0017] Based on the bounding boxes of each logistics target in the BEV plane and the appearance features, a composite similarity metric function is constructed to calculate the similarity between each logistics target;

[0018] Based on the similarity between logistics targets, the DBSCAN algorithm is used to cluster logistics targets.

[0019] The bounding boxes and corresponding appearance features of the logistics targets closest to each cluster center are used as the deduplicated bounding boxes and corresponding appearance features of each logistics target in the BEV plane.

[0020] Furthermore, the composite similarity metric function is:

[0021]

[0022] Where p and q represent the logistics targets whose similarity is to be calculated; and Let f represent the bounding boxes of logistics target p and logistics target q in the BEV plane, respectively; p and f q X represents the appearance characteristics of logistics target p and logistics target q, respectively; p and X q Let represent the center points of the bounding boxes of logistics targets p and q in the BEV plane, respectively; α represents the intersection-union ratio weight; β represents the weight coefficient of the cosine similarity of appearance features; γ represents the weight coefficient of the distance decay term; and σ represents the distance decay coefficient.

[0023] Furthermore, based on the bounding boxes and corresponding appearance features of each logistics target in the BEV plane after deduplication in each frame of BEV images, the DeepSORT target tracking algorithm is used to track the targets, and the tracking results of each logistics target in each frame of BEV images are obtained.

[0024] Furthermore, the target detection information for the logistics target includes the bounding box, confidence level, and category of the logistics target;

[0025] The target tracking results for the logistics target include tracking ID, center point BEV coordinates, velocity vector, trajectory survival frame count, and trajectory status.

[0026] Furthermore, based on the target tracking results, obtaining the position and status of each logistics target in the work area at each frame time point includes:

[0027] The work area layout map is mapped onto each frame of BEV image to obtain the coordinate range of each logistics partition in the work area in the BEV image; wherein, the logistics partition includes cross-area area, stack area area and roller conveyor area;

[0028] Based on the BEV coordinates of the center point of each logistics target in each frame of BEV image and the coordinate range of each logistics partition in the BEV image, the logistics partition where each logistics target is located in each frame of BEV image is obtained.

[0029] Based on the logistics partition where each logistics target is located in each frame of BEV image, the center point BEV coordinates and velocity vector of each logistics target in each frame of BEV image and the corresponding previous frame of BEV image, the state of each logistics target in each frame of BEV image is obtained.

[0030] Furthermore, obtaining the state of each logistics target in each frame of the BEV image includes:

[0031] Based on the BEV coordinates of the center point of each logistics target in each frame of BEV image and the corresponding previous frame of BEV image, the position change of each logistics target is obtained.

[0032] Based on the velocity vectors of each logistics target in each frame of BEV image and the corresponding previous frame of BEV image, the velocity change and velocity change direction of each logistics target are obtained.

[0033] When the position change and speed change of each billet and each crane are both within the preset threshold, the state of each billet and each crane is stationary.

[0034] Otherwise, when the BEV coordinate distance between the center points of the billet and the trolley is less than or equal to a preset threshold, and the speed change of the billet and the trolley is in the same direction, the billet is in the state of being hoisted and the trolley is in the state of strip steel trolley; otherwise, the trolley is in the state of moving empty; when the billet is located in the roll area, the billet is in the state of moving; when the billet is located outside the roll area, the billet is in the state of abnormal movement.

[0035] Furthermore, the method also includes performing spatiotemporal calibration on each camera in the work area before acquiring the monitoring video captured by each camera set in the work area.

[0036] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0037] 1. This invention reduces target occlusion blind spots by arranging multiple cameras in the work area to cover different perspectives, ensuring full coverage of the target in complex scenes. The target detection boxes detected by multiple cameras at the same time are mapped onto the BEV plane through inverse perspective transformation, which reduces the complexity caused by target occlusion and perspective changes, making target tracking more accurate and stable, and providing a global perspective of logistics.

[0038] 2. When deduplicating logistics targets after multiple camera captures, this invention combines the ReID model to extract the appearance features of the targets, and further improves the robustness of target detection and tracking by constructing a composite similarity metric function and clustering algorithm to deduplicate targets.

[0039] 3. This invention uses the Mamba model to detect steel billets and overhead cranes in images. The Mamba model's efficient matching ability can quickly and accurately identify and track targets in dynamic scenes, while also possessing high robustness, maintaining stable performance under conditions of changing lighting, target diversity, and real-time requirements in dynamic environments.

[0040] 4. This invention maps the work area layout map onto the BEV image of target tracking to obtain the global location information and status of each logistics target in the entire work area, enabling managers to have a comprehensive understanding of the dynamic distribution of logistics targets, thereby better carrying out production scheduling and resource management.

[0041] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0042] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0043] Figure 1 This is a flowchart illustrating a steel logistics tracking method based on the Mamba model and BEV in an embodiment of the present invention.

[0044] Figure 2 This is a schematic diagram of the BEV (Balance of Vehicle) of the working area in an embodiment of the present invention. Detailed Implementation

[0045] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0046] A specific embodiment of the present invention discloses a steel logistics tracking method based on the Mamba model and BEV, such as... Figure 1 As shown, it includes the following steps S1-S4:

[0047] Step S1: Obtain the monitoring video captured by each camera set in the work area, and extract frames according to a preset time interval to obtain the sampling frame sequence of each camera at the same time point.

[0048] Specifically, a heterogeneous multi-source visual sensor network architecture is adopted, and several high-resolution industrial-grade cameras are deployed to build a three-dimensional visual perception network covering the entire operation area. This includes cameras set at both ends of the roller conveyor and several cameras evenly distributed in the storage area. The cameras set at both ends of the roller conveyor can monitor the movement of steel billets on the roller conveyor. The cameras set in the storage area are used to monitor the storage and handling of steel billets in the storage area, covering every corner of the storage area to reduce blind spots.

[0049] Furthermore, before acquiring the monitoring videos captured by each camera set up in the work area, spatiotemporal calibration is performed on each camera in the work area.

[0050] Specifically, spatiotemporal calibration involves calibrating the spatial position and temporal synchronization of cameras to ensure that images captured by multiple cameras can be accurately mapped to the same coordinate system in space, while maintaining temporal synchronization.

[0051] More specifically, a checkerboard calibration method is used to obtain the intrinsic and extrinsic parameter matrices of each camera, establishing a rigid transformation relationship between a unified world coordinate system and the coordinate systems of each camera. Through spatial calibration, two-dimensional images captured by different cameras can be mapped to the same coordinate system, thereby achieving multi-view image fusion. During time synchronization calibration, the PTP protocol is used to achieve millisecond-level time synchronization among the cameras.

[0052] Furthermore, in the surveillance video, a frame is extracted at fixed time intervals to form a sampling frame sequence, ensuring that all cameras extract frames at the same time point, thereby obtaining an image sequence of each camera that is synchronized in time.

[0053] It should be noted that in steel logistics, the speed of steel billets conveyed on roller conveyors is usually a few centimeters to tens of centimeters per second; when steel billets are hoisted by overhead cranes, their speed is usually tens of centimeters to several meters per second. Therefore, in this embodiment, 10 frames are extracted per second, that is, frames are extracted at a time interval of 0.1s to obtain the sampling frame sequence of each camera at the same time point.

[0054] Step S2: Perform target detection on each frame image in each of the sampled frame sequences using the trained Vision Mamba model to obtain target detection information for each logistics target in each frame image; wherein, the logistics targets include steel billets and overhead cranes.

[0055] Specifically, the Vision Mamba model is a deep learning object detection model based on a convolutional neural network architecture, used to efficiently and accurately detect and locate objects in complex scenes.

[0056] When training the Vision Mamba model, it was first pre-trained on the ImageNet-1K dataset to optimize the model's general feature extraction capabilities. Then, it was fine-tuned using the COCO 2017 dataset and a self-built steel logistics dataset that incorporated images of billets, overhead cranes, and their respective locations and categories to improve the model's generalization ability in object detection tasks and its adaptability to specific scenarios.

[0057] Specifically, a phased training strategy is adopted to train the Vision Mamba model, including:

[0058] During the pre-training phase, the backbone network of the Vision Mamba model was trained using the large-scale general dataset ImageNet-1K. The parameters of the detection head were frozen, and the cross-entropy loss function was used for training. The loss function was minimized through backpropagation and the Adam optimization algorithm. The parameters of the backbone network were saved after 50 epochs of pre-training.

[0059] In the mixed data training phase, based on pre-training, the Vision Mamba model is trained using the COCO 2017 dataset and the self-built steel logistics dataset. The total loss function is the sum of the focus loss function, the complete intersection-union ratio loss function (CIoULoss), and the centrality prediction loss function (BCE Loss). The loss function is minimized through backpropagation and the Adam optimization algorithm. After training for 100 epochs, the trained Vision Mamba model is obtained.

[0060] During the fine-tuning phase in the industrial scenario, only the self-built steel logistics dataset is used to increase the resolution of the input images to 800×800. Simultaneously, the shallow layer parameters of the Vision Mamba model's backbone network are frozen; that is, the parameters of the layers extracting common low-level features in the Vision Mamba model are frozen. For example, the parameters of the first four layers of the backbone network are frozen, and the learning rate of the training detection head and the Bi-SSM high-level modules is reduced to 1×10⁻⁶. -4 Furthermore, the online hard sample mining (OHEM) method was used to select the top 30% of samples with the highest loss values ​​for training, in order to improve the ability to distinguish dense steel billets. After training for 100 epochs, the final trained Vision Mamba model was obtained.

[0061] Furthermore, the target detection information for the logistics target includes the bounding box, category confidence, and category of the logistics target.

[0062] Specifically, the bounding box of the logistics target represents its position in the image, using four coordinates (x, y, z). min x max y min y max The category of the logistics target is either steel billet or overhead crane.

[0063] Step S3: Map the target detection information of each logistics target in the images at the same time point in each of the sampling frame sequences to the same BEV plane to obtain the BEV image corresponding to each time point, forming a BEV frame sequence.

[0064] Specifically, the BEV plane, also known as the Bird's Eye View (BEV), is a two-dimensional plane that represents the view from above. On this plane, target information detected by different cameras at the same time can be fused to form a global view.

[0065] Furthermore, using a pre-trained ReID model, the appearance features of each logistics target in each frame image are extracted.

[0066] Specifically, the appearance features of each logistics target include, but are not limited to, color features, texture features, and shape features. The ReID (Re-Identification) model is used to extract the appearance features of the target to help the system identify the appearance of the same target in different frames during target tracking tasks. In this embodiment, the OSNet model is used, which can effectively extract robust appearance features through multi-scale feature extraction and attention mechanisms. It can effectively handle target recognition at different scales and viewpoints, thereby improving the accuracy and robustness of target re-identification.

[0067] More specifically, an image dataset containing annotations of steel billets and overhead cranes is used, and after data augmentation and preprocessing, the ReID model is trained. Then, the appearance features of the target are extracted from the trained model for subsequent target matching and tracking.

[0068] Furthermore, the step of mapping the target detection information of each logistics target in the images at the same time point in each of the sampling frame sequences to the same BEV plane includes steps S31-S32:

[0069] Specifically, at each time point, the bounding boxes and appearance features of all logistics targets are mapped onto a unified BEV plane and deduplicated to form a BEV image for that time point. This BEV image contains the global location information of all logistics targets at that time point.

[0070] Step S31: Project the bounding boxes of each logistics target in each frame image at the same time point in the sampling frame sequence onto the same BEV plane to obtain the BEV bounding boxes of each logistics target in the BEV plane.

[0071] Furthermore, the corner points of the bounding boxes of each of the logistics targets are projected onto the BEV plane using inverse perspective mapping to obtain the BEV bounding boxes of each of the logistics targets.

[0072] Specifically, the inverse perspective mapping projection is a method of mapping pixels in an image to corresponding points on the BEV plane; wherein, based on the intrinsic and extrinsic parameter matrices of each camera obtained in step S1, points in the world coordinate system are mapped to the image coordinate system. By solving for the inverse of the rotation matrix and the inverse operation of the translation vector in the intrinsic and extrinsic parameter matrices, the inverse perspective transformation matrix is ​​calculated, and the four corner points (x, y, z) of the bounding box of each logistics target are mapped. min y min ), (x max y min ), (x min y max ), (x max y max Using inverse perspective transformation matrices, the coordinates of the corresponding points of each logistics target on the BEV plane are obtained. The coordinates of the four corner points mapped onto the BEV plane are then used to construct the BEV bounding box using the minimum bounding rectangle method, thus obtaining the BEV bounding box of each logistics target on the BEV plane.

[0073] Step S32: Based on the BEV bounding box of each logistics target and the appearance features extracted from each frame image at the corresponding time point, deduplication is performed on each logistics target to obtain the bounding box and corresponding appearance features of each logistics target in the BEV plane after deduplication at the corresponding time point.

[0074] Specifically, in multi-camera scenarios, the same logistics target may be detected by multiple cameras simultaneously, or there may be false detections or missed detections. By calculating similarity, duplicate targets can be identified, and clustering algorithms can be used to merge them into a unique representation.

[0075] Furthermore, the process of deduplicating each of the logistics targets to obtain the bounding box and corresponding appearance features of each deduplicated logistics target in the BEV plane includes steps S321-S323:

[0076] Step S321: Based on the bounding boxes of each logistics target in the BEV plane and the appearance features, construct a composite similarity metric function to calculate the similarity between each logistics target.

[0077] Furthermore, the composite similarity metric function is:

[0078]

[0079] Where p and q represent the logistics targets whose similarity is to be calculated; and Let f represent the bounding boxes of logistics target p and logistics target q in the BEV plane, respectively; p and f qX represents the appearance characteristics of logistics target p and logistics target q, respectively; p and X q Let represent the center points of the bounding boxes of logistics targets p and q in the BEV plane, respectively; α represents the intersection-union ratio weight; β represents the weight coefficient of the cosine similarity of appearance features; γ represents the weight coefficient of the distance decay term; and σ represents the distance decay coefficient.

[0080] Specifically, the composite similarity metric function considers spatial similarity, appearance similarity, and distance decay, enabling a comprehensive assessment of the similarity between two logistics targets. Spatial similarity is calculated by determining the intersection-union ratio (IUU) of the bounding boxes of the two logistics targets in the BEV plane, measuring the degree of spatial overlap. Appearance similarity is calculated by using the appearance features obtained in step S3 to determine the cosine similarity of the appearance features between the logistics targets, used to determine whether the two logistics targets are the same target. The distance decay factor calculates the Euclidean distance between the center points of the two logistics targets in the BEV plane. By introducing a distance decay term, targets that are far apart are considered different targets even if they are similar in other aspects, avoiding the erroneous merging of distant targets and ensuring target locality.

[0081] For example, the crossover ratio weight α, the weight coefficient β of the appearance feature cosine similarity, and the weight coefficient γ of the distance attenuation term can be set to 0.3, 0.3, and 0.4, respectively; and the distance attenuation coefficient σ can be set to 0.5.

[0082] Step S322: Based on the similarity between logistics targets, use the DBSCAN algorithm to cluster logistics targets.

[0083] Specifically, the DBSCAN algorithm is a density-based clustering algorithm that can effectively identify clusters in high-density areas. Its core idea is to define clusters based on density. The DBSCAN algorithm divides logistics targets within a range into three categories based on two parameters: radius ∈ and minimum number of targets MinPts. These include core targets, which are targets with at least the minimum number of targets MinPts within radius ∈; boundary targets, which are targets with fewer than the minimum number of targets MinPts within radius ∈ but still belong to a certain cluster; and noisy targets, which are targets that are neither core targets nor boundary targets.

[0084] Based on the similarity between each logistics target, the similarity is converted into the distance between each logistics target using the following formula:

[0085] D(p,q)=1-S(p,q)

[0086] Among them, logistics targets with high similarity are close in distance, while logistics targets with low similarity are far apart.

[0087] Based on the distance between each logistics target, a preset radius ∈, and a preset minimum number of targets MinPts, all logistics targets are initialized to unvisited. An unvisited logistics target is randomly selected, and all logistics targets whose distance to it is less than the preset radius ∈ are searched. When the number of logistics targets within the preset radius ∈ is greater than or equal to the preset minimum number of targets MinPts, the logistics target is marked as a core target, and all logistics targets within the preset radius ∈ are added to the current cluster. For each logistics target within the preset radius ∈ of the core target, the above steps are repeated until no new logistics targets can be added to the current cluster. When the number of logistics targets within the preset radius ∈ is less than the preset minimum number of targets MinPts, if the logistics target belongs to another logistics target cluster, it is set as a boundary target; if the logistics target does not belong to another logistics target cluster, it is set as a noise target. All the above steps are repeated until all logistics targets are marked, resulting in several logistics target clusters.

[0088] Step S323: Use the bounding box and corresponding appearance features of the logistics target closest to each cluster center as the deduplicated bounding box and corresponding appearance features of each logistics target in the BEV plane.

[0089] Specifically, for each logistics target cluster, the logistics target closest to the current cluster center is selected as the real logistics target and its appearance features are retained. Other logistics targets in the current cluster are removed to obtain the bounding box and corresponding appearance features of each logistics target in the BEV plane after deduplication.

[0090] It should be noted that an appropriate radius ∈ is selected based on the distribution density of the logistics targets; for example, it can be set to 0.5. For the minimum number of targets MinPts, for example, it can be set to 4 or 5, representing the minimum number of neighbors required for a logistics target to become a core target.

[0091] By using similarity-based DBSCAN clustering, duplicate logistics targets can be effectively removed. It not only considers the spatial location and appearance features of the targets, but also processes noise and outliers through density clustering, ensuring the uniqueness of each target in the BEV plane and improving the accuracy and robustness of target deduplication and tracking.

[0092] More specifically, the BEV images generated at each time point are arranged in chronological order to form a BEV frame sequence, which provides a globally consistent basis for subsequent target tracking and state analysis.

[0093] Step S4: Based on the target detection information of each logistics target in each frame of the BEV image of the BEV frame sequence, target tracking is performed on each logistics target in each frame of the BEV image, and the position and status of each logistics target in the work area at each frame time point are obtained based on the target tracking results.

[0094] Furthermore, based on the bounding boxes and corresponding appearance features of each logistics target in the BEV plane after deduplication in each frame of BEV images, the DeepSORT target tracking algorithm is used to track the targets, and the tracking results of each logistics target in each frame of BEV images are obtained.

[0095] Specifically, the DeepSORT target tracking algorithm is a multi-target tracking algorithm based on Kalman filters and appearance features. It combines the bounding box information of target detection with appearance features, predicts the target's trajectory through Kalman filters, and uses appearance features for data association, thereby achieving continuous target tracking.

[0096] Furthermore, the target tracking results for the logistics target include tracking ID, center point BEV coordinates, velocity vector, trajectory survival frame count, and trajectory status.

[0097] Specifically, when tracking each of the logistics targets, the DeepSORT tracker is initialized. For the first frame of the BEV image, all detected logistics targets are initialized as new trajectories with their tracking IDs set. Their initial states include bounding boxes and appearance features. In subsequent frames of the BEV image, the DeepSORT algorithm updates existing trajectories based on the detected targets. For each existing trajectory, a Kalman filter is used to predict the position of each logistics target in each frame of the BEV image, and the predicted position of each logistics target and the appearance features of the current trajectory are matched with the detected logistics targets in each frame. The matching includes motion matching and appearance matching. Motion matching measures the similarity in position based on the Mahalanobis distance between the predicted position of each logistics target and its position in each frame of the BEV image. Appearance matching measures the similarity in appearance based on the cosine similarity between the appearance features of each logistics target on the current trajectory and the appearance features of each logistics target in each frame of the BEV image.

[0098] It should be noted that for each logistics target detected in each frame of BEV images, when its confidence level is greater than a preset high confidence threshold (for example, the high confidence threshold is set to 0.7), a weighted cost is calculated using motion matching and appearance matching for each trajectory and each logistics target detected in each frame of BEV images; for example, the weight of motion matching is set to 0.8 and the weight of appearance matching is set to 0.2. The Hungarian algorithm is used to complete the trajectory association between each trajectory and each logistics target detected in each frame of BEV images with a confidence level greater than the preset threshold based on the calculated weighted costs, ensuring that high-confidence targets can be accurately assigned to the corresponding trajectories.

[0099] When the confidence level of each logistics target detected in each frame of unmatched BEV images is greater than the medium confidence threshold, for example, the medium confidence threshold is set to 0.3. For unmatched trajectories and each logistics target detected in each frame of BEV images, only motion matching is used to complete the trajectory association of the medium confidence logistics targets.

[0100] Based on the matching results above, the state of the Kalman filter is updated using the bounding box of the detected target, and the appearance features of the trajectory are updated using the appearance features of the detected target; wherein, the appearance features of the trajectory are updated by using a sliding window mechanism, retaining the appearance features of the most recent several frames, and calculating their average as the appearance features of the trajectory; for example, the length of the sliding window can be set to 20 frames.

[0101] More specifically, for two consecutive frames where a logistics target detection result fails to match, if its confidence level is greater than a preset new trajectory confidence threshold (for example, the new trajectory confidence threshold can be set to 0.5), it is initialized as a new trajectory with a "Tentative" status. This allows the algorithm to further observe and confirm these trajectories in subsequent frames, avoiding premature deletion of valid trajectories due to false detections or occlusion. When a trajectory in the "Tentative" state successfully matches a logistics target detection result in three consecutive frames, its status is changed to "Confirmed," indicating that the trajectory has been stably detected in several consecutive frames and can be used for subsequent tracking and analysis. When a trajectory in the "Confirmed" state fails to match a logistics target detection result for several consecutive frames, the trajectory is deleted; for example, this can be set to fail to match a logistics target for 30 consecutive frames.

[0102] It should be noted that the center point BEV coordinates of the target tracking result are calculated based on the coordinates of the four corner points of the target detection box in each frame of the BEV image; the velocity vector of the target tracking result is obtained by predicting and updating the motion state of the target through a Kalman filter.

[0103] Furthermore, based on the target tracking results, obtaining the position and status of each logistics target in the work area at each frame time point includes steps S41-S43:

[0104] Step S41: Map the work area layout map onto each frame of BEV image to obtain the coordinate range of each logistics partition in the work area in the BEV image; wherein, the logistics partition includes cross-area area, stack area area and roller conveyor area.

[0105] Specifically, such as Figure 2 As shown, the work area layout diagram is typically a two-dimensional diagram used to describe the location and shape of different functional areas within the work area. In a steel processing environment, the work area is divided into multiple "spans," each of which can be considered an independent work area for a specific production or storage task; stacking areas are areas for storing materials such as steel billets; and roller conveyors are areas where materials such as steel billets move on roller conveyors. In this embodiment, the work area is divided into 14 spans distributed along the X-axis, 51 stacking areas distributed along the Y-axis, and a roller conveyor area between the 33rd and 34th stacks. More specifically, in the mapped BEV image, the boundaries of each span, stacking area, and roller conveyor area are converted into corresponding rectangular coordinate ranges.

[0106] Step S42: Based on the BEV coordinates of the center point of each logistics target in each frame of BEV image and the coordinate range of each logistics partition in the BEV image, obtain the logistics partition where each logistics target is located in each frame of BEV image.

[0107] Specifically, for the BEV coordinates of the center point of each logistics target in each frame of the BEV image, the location of the logistics target is determined by checking which logistics zone it falls into. It should be noted that in this embodiment, each span contains a certain number of stacking areas, allowing the storage and processing of billets in different spans, while maintaining the span numbering for management and scheduling.

[0108] Step S43: Based on the logistics partition where each logistics target is located in each frame of BEV image, the center point BEV coordinates and velocity vector of each logistics target in each frame of BEV image and the corresponding previous frame of BEV image, obtain the state of each logistics target in each frame of BEV image.

[0109] Specifically, for steel billets, their states include stationary, moving, being hoisted, and abnormal movement; for overhead cranes, their states include stationary, moving without load, and moving strip steel.

[0110] Furthermore, obtaining the state of each logistics target in each frame of the BEV image includes:

[0111] Based on the BEV coordinates of the center point of each logistics target in each frame of BEV image and the corresponding previous frame of BEV image, the position change of each logistics target is obtained.

[0112] Based on the velocity vectors of each logistics target in each frame of BEV image and the corresponding previous frame of BEV image, the velocity change and velocity change direction of each logistics target are obtained.

[0113] Specifically, the velocity of the j-th logistics target in the i-th frame of the BEV image can be expressed as: The speed of the logistics target in the previous BEV image can be represented as Therefore, the change in velocity can be expressed by the following formula:

[0114]

[0115] More specifically, the direction of the velocity change is calculated based on the amount of velocity change; wherein, depending on the sign of the velocity change, when At that time, the speed of the logistics target changes in the rightward direction; when At that time, the direction of change in the speed of the logistics target is to the left; when At that time, the direction of change in the speed of the logistics target is upward; when At that time, the direction of change in the speed of the logistics target is downward.

[0116] The following formula is used to calculate the angle between the velocity change of the logistics target and the x-axis, thus obtaining a more accurate velocity direction of the logistics target:

[0117]

[0118] Furthermore, when the positional and velocity changes of each billet and each crane are both within a preset threshold, the billets and cranes are in a stationary state.

[0119] Otherwise, when the BEV coordinate distance between the center points of the billet and the trolley is less than or equal to a preset threshold, and the speed change of the billet and the trolley is in the same direction, the billet is in the state of being hoisted and the trolley is in the state of strip steel trolley; otherwise, the trolley is in the state of moving empty; when the billet is located in the roll area, the billet is in the state of moving; when the billet is located outside the roll area, the billet is in the state of abnormal movement.

[0120] Specifically, when the BEV coordinate distance between the center points of the billet and the overhead crane is less than or equal to a preset threshold, and the speed change of the billet and the overhead crane is in the same direction, it indicates that the billet and the overhead crane are moving simultaneously. When the overhead crane moves, and there is no billet moving in the same direction and at the same speed near its center point, it indicates that the overhead crane is running unloaded. When the billet moves along the roller conveyor in the roller area, and there is no overhead crane moving in the same direction and at the same speed near its center point, it indicates that the billet is being transported by the roller conveyor. When the billet is outside the roller area, and there is no overhead crane moving in the same direction and at the same speed near its center point, it indicates that the billet is moving abnormally, and an alarm should be triggered to prompt staff to conduct an on-site inspection.

[0121] For example, the preset threshold for the position change can be set to 0.01 meters; the preset threshold for the speed change can be set to 0.01 meters per second; the preset threshold for the BEV coordinate distance between the center point of the billet and the overhead crane can be set to 0.5 meters; and the threshold for determining whether the speed change direction is consistent can be set to θ < 10°.

[0122] In summary, the steel logistics tracking method based on the Mamba model and BEV of this invention has the following beneficial effects:

[0123] 1. This invention reduces target occlusion blind spots by arranging multiple cameras in the work area to cover different perspectives, ensuring full coverage of the target in complex scenes. The target detection boxes detected by multiple cameras at the same time are mapped onto the BEV plane through inverse perspective transformation, which reduces the complexity caused by target occlusion and perspective changes, making target tracking more accurate and stable, and providing a global perspective of logistics.

[0124] 2. When deduplicating logistics targets after multiple camera captures, this invention combines the ReID model to extract the appearance features of the targets, and further improves the robustness of target detection and tracking by constructing a composite similarity metric function and clustering algorithm to deduplicate targets.

[0125] 3. This invention uses the Mamba model to detect steel billets and overhead cranes in images. The Mamba model's efficient matching ability can quickly and accurately identify and track targets in dynamic scenes, while also possessing high robustness, maintaining stable performance under conditions of changing lighting, target diversity, and real-time requirements in dynamic environments.

[0126] 4. This invention maps the work area layout map onto the BEV image of target tracking to obtain the global location information and status of each logistics target in the entire work area, enabling managers to have a comprehensive understanding of the dynamic distribution of logistics targets, thereby better carrying out production scheduling and resource management.

[0127] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A steel logistics tracking method based on the Mamba model and BEV, characterized in that, Includes the following steps: Acquire the monitoring video captured by each camera set in the work area, and extract frames at preset time intervals to obtain the sampling frame sequence of each camera at the same time point; The trained Vision Mamba model is used to perform target detection on each frame image in each of the sampled frame sequences to obtain target detection information for each logistics target in each frame image; wherein, the logistics targets include steel billets and overhead cranes; The target detection information of each logistics target in the images at the same time point in each of the sampling frame sequences is mapped to the same BEV plane to obtain the BEV image corresponding to each time point, forming a BEV frame sequence. Based on the target detection information of each logistics target in each frame of the BEV frame sequence, target tracking is performed on each logistics target in each frame of the BEV image, and the position and status of each logistics target in the work area at each frame time point are obtained based on the target tracking results.

2. The method according to claim 1, characterized in that, It also includes using a pre-trained ReID model to extract the appearance features of each of the logistics targets in each frame of the image; The step of mapping the target detection information of each logistics target in the images at the same time point in each of the sampling frame sequences to the same BEV plane includes: Project the bounding boxes of each logistics target in each frame image at the same time point in the sampling frame sequence onto the same BEV plane to obtain the BEV bounding boxes of each logistics target in the BEV plane. Based on the BEV bounding boxes of each logistics target and the appearance features extracted from each frame image at the corresponding time point, the logistics targets are deduplicated to obtain the bounding boxes and corresponding appearance features of each logistics target in the BEV plane after deduplication at the corresponding time point.

3. The method according to claim 2, characterized in that, The step of obtaining the BEV bounding box of each of the logistics targets on the BEV plane includes: projecting each corner point of the bounding box of each of the logistics targets onto the BEV plane using inverse perspective mapping to obtain the BEV bounding box of each of the logistics targets.

4. The method according to claim 3, characterized in that, The process of deduplicating each of the logistics targets to obtain the bounding box and corresponding appearance features of each deduplicated logistics target in the BEV plane includes: Based on the bounding boxes of each logistics target in the BEV plane and the appearance features, a composite similarity metric function is constructed to calculate the similarity between each logistics target; Based on the similarity between logistics targets, the DBSCAN algorithm is used to cluster logistics targets. The bounding boxes and corresponding appearance features of the logistics targets closest to each cluster center are used as the deduplicated bounding boxes and corresponding appearance features of each logistics target in the BEV plane.

5. The method according to claim 4, characterized in that, The composite similarity metric function is: Where p and q represent the logistics targets whose similarity is to be calculated; and Let f represent the bounding boxes of logistics target p and logistics target q in the BEV plane, respectively; p and f q X represents the appearance characteristics of logistics target p and logistics target q, respectively; p and X q Let represent the center points of the bounding boxes of logistics targets p and q in the BEV plane, respectively; α represents the intersection-union ratio weight; β represents the weight coefficient of the cosine similarity of appearance features; γ represents the weight coefficient of the distance decay term; and σ represents the distance decay coefficient.

6. The method according to any one of claims 1-5, characterized in that, Based on the bounding boxes and corresponding appearance features of each logistics target in the BEV plane after deduplication in each frame of BEV image, the DeepSORT target tracking algorithm is used to track the targets and obtain the tracking results of each logistics target in each frame of BEV image.

7. The method according to claim 1, characterized in that, The target detection information for logistics targets includes the bounding box, confidence level, and category of the logistics targets; The target tracking results for the logistics target include tracking ID, center point BEV coordinates, velocity vector, trajectory survival frame count, and trajectory status.

8. The method according to claim 7, characterized in that, The step of obtaining the position and status of each logistics target in the work area at each frame time point based on the target tracking results includes: The work area layout map is mapped onto each frame of BEV image to obtain the coordinate range of each logistics partition in the work area in the BEV image; wherein, the logistics partition includes cross-area area, stack area area and roller conveyor area; Based on the BEV coordinates of the center point of each logistics target in each frame of BEV image and the coordinate range of each logistics partition in the BEV image, the logistics partition where each logistics target is located in each frame of BEV image is obtained. Based on the logistics partition where each logistics target is located in each frame of BEV image, the center point BEV coordinates and velocity vector of each logistics target in each frame of BEV image and the corresponding previous frame of BEV image, the state of each logistics target in each frame of BEV image is obtained.

9. The method according to claim 8, characterized in that, The process of obtaining the state of each logistics target in each frame of the BEV image includes: Based on the BEV coordinates of the center point of each logistics target in each frame of BEV image and the corresponding previous frame of BEV image, the position change of each logistics target is obtained. Based on the velocity vectors of each logistics target in each frame of BEV image and the corresponding previous frame of BEV image, the velocity change and velocity change direction of each logistics target are obtained. When the position change and speed change of each billet and each crane are both within the preset threshold, the state of each billet and each crane is stationary. Otherwise, when the BEV coordinate distance between the center points of the billet and the trolley is less than or equal to a preset threshold, and the speed change of the billet and the trolley is in the same direction, the billet is in the state of being hoisted and the trolley is in the state of strip steel trolley; otherwise, the trolley is in the state of moving empty; when the billet is located in the roll area, the billet is in the state of moving; when the billet is located outside the roll area, the billet is in the state of abnormal movement.

10. The method according to claim 1, characterized in that, The method further includes performing spatiotemporal calibration on each camera in the work area before acquiring the monitoring video captured by each camera set in the work area.

Images